Cave Markings 

I have been moving quickly between tasks since my return from Boston.  I'm finally taking some time off for the Independence Day holiday.  Floating in the pool, I keep finding the subject of Ted Neward's essay coming to mind.  It appears I still have something I want to say about ORM.  In this series of  posts, I hope to explain why Visual FoxPro has more mojo for resolving Object Relational Mapping issues than any other technology, of which I am aware.

At the speaker's dinner for the Atlanta Code Camp, I discussed LINQ with Charlie Arehart, and Wally.  I brought up the doomed Object Spaces project, and quoted Ted Neward's pithy line that "Object relational mapping is the Vietnam of computer science."  It is a good line.  It is also true, as much as a pithy metaphor can be true.

Now that Ted has expanded his metaphor, we can see the limitations of metaphor in general.  I recommend Jeff Atwood's analysis of Ted's essay.  I agree with everything Jeff has written there.  Primarily, I agree that Ted tried too hard to stretch his Vietnam analogy which distracted from his primary, quite sound, thesis that "there is no good solution to the object/relational mapping problem."  Full stop.

Despite any over-reaching in support of his Vietnam analogy,  I must state clearly that Ted Neward is possibly the most qualified person in the industry to make the criticisms he does of ORM.  Here's a short bio with his credentials.  Suffice it to say that he is not arguing from a position of ignorance.  Ted works with many technologies, and has seen the limitations of ORM manifested in different languages and platforms.

To continue with my love letter to Ted, he acknowledged something that few people in the industry are even aware of.  Visual FoxPro eases the dissonance between objects and relational data better than any other technology on the planet.  I first came to this realization upon reading this post by Joel Spolsky in 2004.  Joel identified the impedance mismatch between objects and relational data as a shortcoming in all modern programming languages.  Joel didn't bother listing VFP among "modern high-level programming languages."

I'm not going to bitch and moan about the low profile of VFP.  I know what it can do, and it's been earning me a decent living for years.  I just can't believe other languages have not done a better job of "borrowing" the best of what VFP does.  I'm damn tired of all the LISP articles on Reddit telling me how I'm missing out because my language of choice doesn't have feature foo which LISP has had since the '60's.  I also take issue with Paul Graham's essay about the RAD nature of LISP for startup companies.  Any experienced VFP developer could easily make the same arguments in favor of the Fox.

When I first saw Ted's mojo comment, I felt vindicated.  I made a quick post, and forwarded the link to "the Craigs".  I secretly hoped they would write this argument for me.  Fortunately, I have to sort this one out for myself.  I am currently developing an object interface to encapsulate a VFP cursor.  This is totally unnecessary in Visual FoxPro, but I need to leverage my object model from .NET, so I must encapsulate all the tasty VFP goodness behind a COM interface.  This is the ultimate test of VFP's ORM mojo in my experience to date. 

It is my contention that the "secret sauce" to VFP's ORM mojo is the cursor.  A VFP cursor is a data structure like no other.  Most non-VFP developers associate VFP with the DBF storage format.  People seem to assume that it is "just" a database.  Visual FoxPro does have a very powerful in-process data engine which includes a wickedly fast ISAM storage format, but it can use data from any source as a cursor with that same data engine.  This is where we find the secret sauce. 

Other languages need to map data from relational sources to primitive types such as arrays or dictionaries (collections).  The ADO recordset is actually an object oriented implementation of VFP's cursor engine.  The ADO.NET dataset takes this further to include relations in the definition.  Strongly typed datasets are a particularly easy way to handle data in managed code.  The advantage that VFP has is that it handles data in cursors natively.  That is what it is made to do.  Handling data in a columnar format, and setting relations between these table-like data structures is the core of the VFP mission.

VFP developers use SQL commands as first class citizens of the language.  They are handled by the runtime in the same manner as conditional and looping statements such as IF and FOR.  There is no conceptual difference to a VFP developer between addressing a local variable, a property on an object, or a field in a cursor. 

Let's pause for a moment and refine our terms.  Here is what Whil Hentzen has to say about VFP cursors in his excellent (though out of print) introduction to VFP development Fundamentals: Building Visual Studio Applications on a Visual FoxPro 6.0 Foundation:

We’ve been using the term "result set" to refer to the collection of records created by the execution of a SELECT command. While Visual FoxPro is "record oriented" in that we can move from record to record as we desire, other languages that use SQL do not have this capability and can only address a block of records as a single entity. As a result, all SQL operations are oriented toward manipulating this "set of records." They refer to this block of records as a "cursor," which stands for CURrent Set Of Records. A cursor is a table-like structure held in memory; it overflows to disk as memory is used up. In some cases, such as queries that essentially just filter an existing table (in other words, no calculations or joins), Visual FoxPro opens a second copy of the table in another work area and then applies the filtering condition to present a different view of the table as the query requests it.

Cursors have two advantages. First, they are fast to set up, partly because of the possible filtering of an existing table, and also because of their initial creation in RAM. Second, maintenance is greatly reduced, because when a cursor is closed, any files on disk created due to RAM overflow are automatically deleted.

I hope it is clear from the above that any data that can be understood conceptually as a row/column data source can be handled very elegantly by VFP.  The key point to recognize is that the data runtime and the language runtime are one.  They even share a common garbage collection mechanism, which you can read about here.  VFP developers are able to enumerate, search, sort or perform set based operations against one or more cursors without making a context shift from the surrounding procedural code.

Cursors are not perfect.  They are not objects, and can therefore not be passed around by reference.  There are two workarounds for this conundrum: ADO recordsets, and XML.  I use XML exclusively.  I no longer consider ADO recordsets to be a maintainable solution.  While the shortcomings of XML are well documented, the flexibility is irresistible.  Furthermore, the VFP XmlAdapter class makes it a snap to translate between cursors and XML, and even take and apply diffgrams.  Therefore, I use XML whenever I am passing data across object/process boundaries, and I convert the XML to VFP cursors within my object interfaces. 

In the next installment on this subject, I will address the object component of Object Relational Mapping.  For now, I hope that I have clarified that Visual FoxPro is not "just" a database.  From my perspective, it is primarily a runtime library for the Visual FoxPro language which seamlessly blends typical imperative programming constructs with relational data handling.

++Alan

I have been trying to formulate a response for the OR/M issue brought up by Ted Neward, but he has done it for me:

“And, although I will likely gather some serious heat for saying this, Visual FoxPro may have some of the most interesting "best of both worlds" mojo in the entire language space on this subject.”

http://blogs.tedneward.com/2006/06/27/Thoughts+On+Vietnam+Commentary.aspx

++Alan

The last day of Tech Ed is different from all the others.  Many people have left already, and all the big announcments have been made.  They do schedule some excellent presentations for Friday, so it is worth hanging around for.

I did manage to attend another session Thursday afternoon.  Steve Lasker gave a presentation about occasionally connected systems.  This talk covered the current and future technologies available from Microsoft to deal with moving data offline, then syncronizing with the server when after returning online.  Good session on a very relevant topic.

Thursday night's party was great fun.  I skipped the ball park food, and went to the upper levels where they had a buffet, and beer options beyond Budweiser products.  I met up with my buddy Jim Topp, and we took in the concert, as well as a significant number of Samual Adams Boston Lagers, together.

The opening act was fine, but not inspiring.  Train was excellent.  They are definately a party band.  Along with all their own songs, they did two Zepplin covers, and closed with Aerosmith's "Dream On".  I would definately pay to see them again.

By Friday morning, I had answered, or at least addressed all of my TFS issues.  Before leaving for the party on Thursday, I brought Randy Miller a printout of CMMi KPAs not fullfilled by the MSF Guidance that my CMMi manager put together.  Randy's response was, "You rock!"  I'm looking forward to working with Randy, and Kevin Kelly as we move forward.  One member of the team with whom I didn't get to talk to enough was Noah Coad.  Noah was at the BOF sessions on Tuesday, but we only spoke briefly Wednesday morning. 

My first session on Friday was Architecting Your Own Enterprise Framework with Brian Button.  Brian shared the lessons he learned working on the Enterprise framework in the Patterns and Practices group.  I plan to download the slides when I return to my office, so I can review the wisdom contained therein.  Brian shared what worked for his team, and what didn't.  The most prominent idea that I left with was the "Rule of 3".  If you see a solution implemented three times, then it should go into a framework, but not before.  According to Brian, frameworks are not developed in the heads of designers, but should be extracted from working code.  This guidance basically follows the YAGNI principle.

During a half-hour break, I caught up with /\/\o\/\/ again to ask him about PowerShell resources for developers.  He pointed me to  Windows PowerShell Programmer's Guide  Unfortunately, it is not currently available in a printable format, but this is an exciting find. 

My next session was Rapid Development of Data End-to-End Solutions and How They Work in an N-Tier Model given by Jay Schmelzer. This was the most code-intensive session I saw all week.  Jay dispensed with slides, and built a data access compenent and windows form to connect with Northwind.  Next he migrated his DAL to a webservice, and showed the changes required to maintain full functionality in the application.  Jay is very energetic, and this was a fantastic session.

My final session after lunch was Architecting Applications for a Service-Oriented World.  Beat Schwegler did an admirable job making SOA appear to be a real technology/architecture.  In contrast to the previous session, Beat used slides exclusively to walk us through a conceptual scenario focusing on maintaining existing IT systems investments, while adding additional value by connecting legacy systems through service contracts.  This is the first time someone has convinced me that SOA is more than just a marketing term from Microsoft.

Throughout the conference, the continuous themes were quality and value.  Microsoft is providing tools to insure quality in the software I produce, while it is up to me to insure that I am producing the right software.  If I build a quality product that doesn't meet the needs of the user, then I have not added any value.

++Alan

Rain this morning.   I hope this clears up before the concert at Fenway tonight.  I'm jazzed about seeing Train perform.

Yesterday afternoon was increadably productive.  I attended a session on Developing Custom Process Templates, Work Item Types and Policies by Kevin Kelly.  This session was similar to the presentation by Dennis Minium at Tech Ed 05, but Kevin had some new content when he got the the demos.

Next, I attended a BOF called "Enhance Your Business and Professional Life By Getting The Most Out of the Local Developer Community".  This session totally exceeded my expectations.  The facilitators did an excellent job of correcting my thinking about my user group.  The take-away quote was "running a user group has nothing to do with coding."  The moderators had plenty of tips about how to get people involved in the group, how to vary the agenda, and how to focus on the community building aspects first with technology placing a far second.

My final session of the day was the best so far, and that is saying a lot.  I have found my guru for deploying Team Foundation Server and his name is Sam Guckenheimer.  Sam refers to himself as a coach, but I will be following his sage nuggets of wisdom like a devotee to his master.

Sam gave me a copy of his book, and I read the first 20 pages last night.  This is the guide I intend to use for our road ahead.  The key piece of wisdom he gave me at the end of his session on "Using Metrics to Manage and Troubleshoot Your Projects" was this:  "The heart of CMMi is process improvement."  Sam totally understood the dissonance between developers trying to implement functionality in an iterative manner, and auditors focused on building to spec.

One of the biggest additions to my understanding of TFS at this conference has been the clarifying the difference between the map and the territory.  The spec is a map.  The MS Project file is a map.  These are maps of where we want to go, but the model needs to reflect reality.  With all the rich access to metrics provided by TFS, we can get a clear picture of the territory and adjust our maps to reflect reality.

++Alan

[Update]  The rain has cleared off, so the concert should be lots of fun. 

My first session today was with Rocky Lhotka.  Rocky was actually very kind to datasets, and didn't harp on the lack of encapsulation.  He showed the advantages of isolating code in partial classes in .NET 2.0.  Of course, this being a Rocky Lhotka session, the ultimate solution was shown to be CSLA.NET.  I asked Rocky if his objects implemented the correct behavior when the user presses the escape key, and he responded that they do.  When he pressed the escape key, the expected behavior didn't happen.  I'm glad to know that these situations still happen to someone of Rocky's stature and experience.

I ate lunch with members of the MOM team, and asked them many questions about connecting MOM and TFS.  They pointed me to AVIcode.  I found the AVIcode booth, and talked with them.  I'll be exploring the resources on their website more upon my return.

I have been haunting the PowerShell booth trying to catch up with /\/\o\/\/.  I went to talk to members of the Visual Studio team about the possiblity of a PowerShell add-in for VS.  They've never heard of PowerShell!  They gave me some good tips on how to follow up through the forums and the feedback site.

While I was talking to a member of the VS team, I asked him about the future of Windows Forms, and he introduced me to CrossBow.  This is exactly the kind of solution I like to see Microsoft producing.  My existing code is not made obsolute by a new presentation technology.  Windows Forms lives!

Probably my last session for today was "Visual Studio 2005 Team System and Microsoft Solution Framework: Implementing an Agile or CMMI Process" with Randy Miller.  This was one of my "must attend" sessions.  My challenges in deploying TFS at this point all revolve around how to make use of the MSF guidance and templates.  Randy gave us the model behind MSF, and offered to email chapters of his MSF book as he writes them.  This was an important session for me.

Upon returning to the Technical Learning Center, I made another stab at talking to someone on the Visual Studio team about PowerShell.  I spoke to Kit George, a Program Manager on the CLR Team, and he had never heard of PowerShell!  He gave me his card, and told me he would follow up on this if I eamil him.

The capstone of the day (so far) was some quality time with Sam Guckenheimer.  I sat down with Sam and reviewed a workflow diagram I made after gathering requirements from my QA department.  Sam had some excellent feedback, and thought through the problem fully.  I made notes to share with my team upon my return.  Sam rocks!

I would be remiss if I did not mention the embarassing situation I went through trying to demo problems I was having with notifications to a member of the VSTS team.  He let me use his laptop to remote into my work desktop and demonstrate the problem.  The real problem turned out to be that the problem no longer exists!  I apparantly had not tried the subscription feature since Beta 3.  When it didn't work, I filed it as an issue in need of further investigation.  The Microsoftie (I didn't get his name) was very gracious, and I was very red faced.  In the end, I'm just happy it works.

I attended BOF (Birds of a Feather) sessions until 10:00 last night.  I accidentally attended one on developing on virtual machines, which was actually informative, if unexpected.  Next was a very useful session on MSF, where the moderator directed me to David Anderson's Blog for more information.  He also suggested that I Google for "Agile CMMI". 

The final session was on migrating to Team Foundation Server and was moderated by Jean-Luc David.  It turns out that Jean-Luc spoke at the Atlanta Code Camp also.  It is odd that we missed each other there.  There was some lively discussion about branching procedures in this BOF.  I enjoyed it completely, and was reassured that we are adopting TFS in a recommended manner at my employer.

I made it to the jam session last night, and didn't get back to my room until 1:30, but I'm feeling fit this morning and ready to take in more tech goodness.

++Alan

[Update] My first session on "Applying Version Control, Work Item Tracking and Team Build to Your Software Development Project" with Brian Harry, was a big disappointment.  I was expecting some prescriptive guidance for integrating these tools into an existing project.  Instead, I sat through the same presentation I saw last year at Tech Ed.  This introductory/marketing style session are tiresome.  The companion session this afternoon promises to dig into customizing the system.  We shall see, I suppose.

At lunch I finally hooked up with Randy Miller.  We had tried to get together yesterday, but he had a conference call run way late.  Randy was very interested in the issues my company is having in resolving the impedence mismatch between the CMMI KPAs and the MSF process guidance.  He took careful notes, and thanked me frequently for providing detailed feedback.  He promised to put me in touch with David Anderson, who owns the CMMI template in TFS.  Randy assured me that David would be interested in following our progress and assisting us with any problems.  This is why I came to Tech Ed!

The first session I tried to attend was full!  I can usually walk into a session late and sit on the front row.   Apparantly, LINQ is a popular topic.  I'll catch the replay on the DVD.

I did attend a session on "Evolving to Patterns" which was actually refactoring to patterns.  I discovered that something I've been doing for years is called Dependency Injection.  It's nice to be assured I'm on the right track. 

The speaker recommended Working Effectively with Legacy Code, which looks like a useful text for my current job.  He also liked Refactoring to Patterns.  I plan to check them both out.

++Alan

[Update] I met Josh Holmes at breakfast this morning.  I also got the skinny from Steve about a cool cigar bar here in Boston.   Steve is actually blogging again, so check it out.

I had a good, long talk with two members of the SQL Everywhere team this morning.  They were very interested in my concerns about FoxPro interoperability.  I had to take some time to explain the VFP development model, but they listened carefully.  The biggest roadblock I see at this point, is the lack of ODBC connections in SQL Everywhere.  They do support OLEDB connections, but the native SQLCONNNECT() function uses ODBC.  I will download the CTP and play with it when I get home.  I promised to post my feedback to the forum for their benefit.

The BCEC is enourmous and attractive, but it is difficult to navigate.  It is sometimes impossible to get from A to B.  Furthermore, the bathrooms are totally inadequate.  This is a modern building, and I expect an abundance of facilities!

[Update2] I got some time in with Ken Levy, and I was blown away.  It turns out that Live is another case of lousy branding by Microsoft.  Live is actually a collection of Web APIs that the team is attempting to normalize.  They will all eventually have common programming models.  In addition, they are implementing javascript libraries with similar programming models, so that a developer can program against a client side javascript object and allow that to converse with the web services.  I was very excited by this demo.

During lunch, I attended a session on managing requirements with Team Foundation Server and MindManager.  Michael Scherotter put together a tool that provides bi-directional interaction with TFS work item tracking.  This is very exciting for visualizing requirement/task dependencies at various levels in the task hierarchy.

[Update3] I just spoke with Steve Lasker about SQL Everywhere and Visual FoxPro.  He is a former VFP developer, and understood my concerns.  He mentioned that he is in regular contact with Calvin Hsia, which put my mind at ease.  Steve also pointed me to this recent blog entry where Calvin demonstrates working with SQL Everywhere.

[Update4] I just finished the first Birds of a Feather session this evening.  Joel Semeniuk led an informative discussion about project management with Team Foundation Server.  I picked up some tricks about configuring MS Project to work properly with TFS.  I'll share the details once I have tried them out.  More BOF sessions to come.

Today was the first "real" day of the conference.  I planned to do too much, and now I'm exhausted. 

Initial impressions: 

Travel:  Boston's airport is bigger than any airport has a right to be. 

The Silver Line is freaking cool!  The articulated bus runs by the airport, then stops at a station where it connects to overhead cables, and the engine shuts off.  Next the bus/trolley travels into its own tunnel.  

Lodging:  Hotel Marlowe in Cambridge is as cool as it looks on the website.  I have no regrets about choosing this funky establishment. 

Food:  There's too damn much of it.  My genetic code tells me to eat when food is available.  Unfortunately, there is always food available.  Breakfast and Lunch were excellent, and the evening reception with free "refreshments" was equally tasty.

People:  I had the opportunity to hang with Eric Sink, Martin Woodward and Don Demask.  I also ran into Matt and Brendan.  I saw Ken Levy, but he was busy pitching Windows Live.  I'll bend his ear tomorrow.  I also ran into Steve Loethen and Drewby.  Steve is my Developer Evangelist, and Drew is the DE in the district north of me.  I hope to get in on some code camps in Drew's district in the coming year.

Technology:  SQL Everywhere looks like a great migration path for users of Jet or Visual FoxPro.  My other sessions where about Team System, and my passions there are well documented.   I'm looking forward to a PowerShell session tomorrow.

I'm off to the jam session. 

++Alan

[Update] No jam session for me.    I went to the hotel to drop off my bag, and the leopard print robe and faux-fur throw looked soooooo comfy.  I was exhausted, and knew that I wouldn't get the most out of Tuesday if I pushed myself too hard.

I got my feedback from the Atlanta Code Camp today.  I admit, I am terrible about providing feedback at events such as this.  Not until today, did I realize how useful those little forms are to the presenter.  So, as a sign of thanks to the people who took the time to provide this data, I want to share it with the world along with my responses.

Score	Comments	My Response
3	more evangelist than teacher	This is possibly accurate, but I could use some examples to back it up.  I'm sorry you didn't get more benefit from your time.
5	(5 lines of comments were blacked out)	Pity, those five lines may have been very helpful.
5
5	very passionate, however demo unorganized, started slowly.  Took 15 minutes to introduce - could have done it in 1.	Accurate. I blew the demo despite my rehearsal. I get excited, then I rush, then I make mistakes. Unfortunately, what the commenter calls the intro was intended to be half the material.
6	A little disappointed because the presenter didn't touch the team system from Architecture point of view.  I think the introduction could have been smaller	Fair. I intentionally ignored the system architecture, and focused on the development process.
7
7	1.  very knowledgeable  2.  as individual programmer/consultant, was not aimed at me	Useful. I work in a team environment, and intended to focus my comments on development teams.
7
7	needed to scroll some of the scripts being shown to the top of the screen so we could see over the heads of those in front	Very helpful. I will definitely keep these comments in mind at my next presentation.
7	well prepared, new insite [sic] on new technology.  Really helpful	Thanks.
7	lots of good information, very applicable	Thank you.
7	wandered a bit. Demo could have been planned better	Useful.  There is a theme here about focus and precise demos.
7	too much generalization	Hmmm… My intent was to back up from the tool, and focus on how it fits into a developer’s work life.
8	good speaker and good intro to the "team" aspects of the team system	Thanks, that’s what I was going for.
9	very good, detail	Much appreciated
9	great speaker, good overview and content	Wow, thank you.
9
9	excellent combination of VS team systems and GTD.  Very informative	Glad you enjoyed it.

My reading of this table is that I need to make sure my demos are bulletproof, and insure visibility for the back rows when showing code.  The most important trend I see here is a need to set clear expectations.

What is fascinating to me, is that I spent a good deal of time at the beginning setting the tone of the presentation, and multiple commenters felt I was wasting time.  My intention was to clearly separate this discussion from a purely technical overview of the tool.  I will be more explicit about this point at my next presentation of this material.

If you attended my session in Atlanta, I would love to hear your comments and suggestions in the comments to this post.  I would also like to hear if anyone thinks I am misreading this data.

This was my first presentation "in the wild."  I've given two talks at the local user group, and many inside my corporate mother ship.  I'm sure that my next presentation will be better as a result of these comments.  I'm equally sure that I will take the time to offer feedback to the speakers at Tech Ed next month.

++Alan

Nothing unique happening today. It's a kid weekend, and if you live your life according to a parenting plan, you know what I mean. For the rest of you, it means the kids are staying at my house this weekend, and I'm loving it.

We live in a terrific neighborhood with a park across the street and lots of playmates as neighbors.  I guess I'm feeling grateful that I can provide this environment for these amazing little humans.  They're doing great, and I'm doing better than I ever have in every aspect of my life.

++Alan