Out of curiosity, I applied the performance tuning change of using char buffers instead of strings to parse the expansion language in the MSS Code Factory Context object.
That's improved performance of the mono release by 50% (down to 22 minutes from 45.)
Presuming that ratio applies to the Java 6 code as well, that means that 35% of the performance improvement of the Java release must be due to JVM differences and/or the results of dynamic object construction and management and collapsing the table records to a single buffer instance.
This is the last update for the 1.7 series, unless I encounter a drastic bug that I need to fix to correct a problem with the 1.8 code. That is very unlikely.
I've gotten MSS Code Factory 1.7 to run under Mono, but it's not quite working the way I want yet. I package a copy of /opt/msscodefactory/cartridge-1.7 under the Properties of the packaging file, but I need to add in some code to search that location instead of using the directory names in the configuration file.
Still, you can do a Subversion checkout of /opt/msscodefactory/cartridge-1.7 to that directory on your system, extract the MSSCodeFactory-1.7.2301-mono.tar.gz, add the resulting directory to your path, and MSSCfCli.exe will run.
After re-reading the GPL, I decided that the attempts to limit the code produced by MSS Code Factory to the GPL was not in compliance, and have removed the code that was enforcing it.
Servlets now establish and release connections. They also initiate transactions and commit them or roll them back as appropriate.
Added isTransactionOpen() and beginTransaction() to ISchema and implementations.
Check for open transactions before and SQL is executed.
Ram implementation pretends there is always a transaction open.
Pg8 implementation tracks an open transaction flag, issuing a "begin transaction" when needed.
Changed the licensing for the rule base to dual commercial-GPLv3. That way, even if someone hacks the code to remove the GPLv3 enforcement, it'll still be illegal to use the code without a commercial license unless you release changes as per the GPL.
Finally got off my butt and set the thread priority lower than foreground processes so the computer doesn't get bogged down by this CPU hog.
Correct some container/component relationships in MssSME, make the use of PIdx for primary index suffixes consistent.
Replicate the template changes to the referencing projects.
The focus for the night was documentation. There are significant updates to the website.
I refactored SchemaPg8BL to SchemaBLPg8 for the sake of consistency. This way all the code you can customize is grouped together.
The logged file names were incorrect. It was annoying me, so I fixed it.
SchemaObj now brings forward the connection methods from its backing store. Servlets now instantiate an ISchemaBLSchemaObj as the session object instead of an ISchemaSchema.
Correct a packaging error.
Sessions are now properly established (tested for GenKb.) Fixed a bug in the engine that was introduced by cleaning up the output logging (it wasn't producing the .xsd files because their file definitions have no srcFileDir attribute.)
It's been a long day of debugging, mostly trying to figure out how to get the Eclipse debugger to work with an external Glassfish instance. Once I got that going, the code debugging went quite quickly.
IMPORTANT: If you're trying to run a Glassfish 3.1 installation as a standalone server as well as under Eclipse for debugging, you need to delete the osgi-cache from the domain directory before starting the server. Otherwise bad data from the other runtime will cause the server startup to fail.
This is a pretty major update -- the exposed interface for the DbIOs that persist the Obj layer have been completely refactored to expose the SchemaBL customized interfaces instead of the underlying Table interfaces returned by the ISchema. This enables direct access to the Business Layer DbIO enhancements.
Fleshed out the initial implementation of the Login servlet to instantiate a PostgreSQL BL schema over the Connection referenced by the jdbc/schemaname JNDI resource.
Implemented servlet Common rules to resolve and verify the existence of the $SchemaName$Schema Session Attribute as an ISchemaBLSchema. If there is any problem doing so, respond with an HTML redirect to the Login form.
As only the SchemaBLRamSchema and SchemaPg8BLSchema can actually be instantiated, they're the only implementations to initialize the table members of a SchemaSchema.
Tidied up the output of the file names to take up less space and to hide the target directory name. In full production, each manufacturing job will be output to a clean temporary directory. After manufacturing, that directory is simply packed into an archive and made available for download by the manufacturing Tenant.
Reformatted the Java 6 code to use a denser coding style. The old style artificially inflated the line counts.
Added isConnected(), connect(), and disconnect() to Schema and implemented in SchemaPg8Schema and SchemaRamSchema.
Added Configuration accessors and JndiName accessors to Schema.
Added constructors to various Schema that take a ConfigurationFile or a String JndiName as parameters. This creates an unconnected Schema, which can then use the connect()/disconnect() semantics rather than being bound to a single connection instance.
The inherited index queries recently added to the DbIOs have been brought forward to the TableObj implementations. This greatly improves the query capabilities of the resulting object interface, and reduces the need for typecasting.
Performance tweak for the license enforcement: cache the host name.
MSS Code Factory is only supposed to manufacture GPLv3 code if it's not running on a licensed node. This is now very crudely and unreliably enforced.
CFCore has been refreshed with the latest MssSME template. All application models should incorporate the MssSME template -- it defines common/standard objects which will be used by the security implementation.
PostgreSQL inserts have been debugged for the most common column data types and work properly.
Reads are partially tested, but not as thoroughly as I'd like yet.
Updates and deletes have not been tested yet.
The 1.8 Java 6 executable testing now successfully loads the complete knowledge base. There are some XML warnings being produced due to remaining errors in the knowledge base XML files, but that's ok for now.
This version not only clean compiles, it can successfully instantiate and persist objects to the Ram database implementation.
This is actually code you could do something with.
Found another bug in the Ram creates. PKey generated values should only be generated during insertion to the base table; subclasses should just use the value that was saved by the base table insert to the buffer.
More debugging. The basics now run pretty good for the Ram layer. I'm spending more time debugging XML parsers than I am manufactured code.
This is a festival of debugging. You can now instantiate a Java Ram database behind a BLObj schema, and use it to create, update, and read objects from the in-memory store. Deletes haven't been tested, but they're pretty straight-forward and should work ok.
Another significant refactoring. The object interface now returns BLObj interfaces, so that application programmers don't have to keep casting objects to the business logic layer. This makes application code a lot cleaner.
There were still some errors in the rules that were exposed by the MSSBam 1.8 model.
I also fixed some bugs in the integrity checking performed by the Ram implementation.
All projects now clean compile.
Interesting that this should be a symbolic build number of 2000...
This major update completes the refactoring from Rec-based storage to more efficient Buff-based storage. Everything clean compiles, but is untested.
Please delete all *Rec.java, *KeyData.java, and *Data.java files from your projects.
The database readDerived methods retrieve anything which derives from this table class. The readBuff methods only return objects which are specifically instances of the table class.
Clean compile, including Ram implementation. However, the Ram implementation is not functional yet.
The old reader APIs need to be reworked as Derived APIs for indexes attached to the table being expanded. For inherited indexes, the table implementing the index is readDerived, and the results need to be filtered for instances derived from this particular subclass.
The Buff methods for the indexes attached to a leaf table just invoke their corresponding Derived APIs and return the results. For tables that have subclasses, the Derived set has to be filtered for instances of this particular subclass (by checking the ClassCode, as the subclasses do implement their inherited interfaces, so you can't use instanceof.)
For inherited Buff readers, the implementing table's readDerived set will be filtered for instances of this particular subclass.
Next on the plate: Reworking the Ram implementation.
The *Rec, *KeyData, and *Data Java objects have been refactored out of existence, and replaced by the *Buff implementations.
The way that the *EditObj and DbIO implementations interact has been cleaned up as well. Instead of relying on the Obj inheritance to trigger a hierarchical invocation of the DbIO methods, the DbIO implementation now performs the cascading operations for object IOs.
The *EditObj layer now just overrides the implementation of create(), update(), and delete() to invoke the appropriately specialized buffer IO implementation.
DbIO implementations for inheritance queries has been completely reworked to minimize the number of trips to the database. Performance should now be roughly linear on the number of sub-classes to be probed rather than linear on the number of objects retrieved.
Clean compile of manufactured code with readDerived and readBuff implemented for PostgreSQL.
Got rid of the TableCache implementations as I've never used them anyhow, so there's no point rewriting them to support the Buffer DbIOs. Please remember to delete all *TableCache.java files from your projects.
The Ram implementation has been stubbed wth "Not implemented yet" exceptions to satisfy the interface.
The object layer will be invoking the readDerived implementations, which combine the results of the subclass readBuff implementations into a collection matching the index query.
Instance readDerived implementations aren't all that efficient, as they need to probe for the ClassCode to determine which table's readBuff to use.
Modified the implementation of the engine to push a new reference context when evaluation popto, poptop, and poponefromtop. This risks creating stack-eating infinite loops with bad rules, but it allows "interesting" rules such as the stack navigations that go on in the Java PostgreSQL layer.
This new functionality has been used to implement the Derived and Buff readers for PostgreSQL.
SchemaPg8TableTable now implements a public final static String providing the core SQL SELECT for populating a SchemaTableBuff.
The unpacking to a buffer is now done in unpackTableResultSetToBuff(), which also invokes the factory constructor for new buffers.
Added the configuration cartridges net-sourceforge-msscodefactory-generator-17-layered+msscf and net-sourceforge-msscodefactory-generator-17-layered+msscf+bl.
I managed to split off the core of my MSS Code Factory engine:
I also modified the engine heavily so that it can be instantiated as many times as you want, instead of there only being one static shared instance. I needed to do this to be able to take advantage of nice fat SMP servers in production. Cheesy
I've partially automated the binding of the engine to any application schema, so it can be used as a formatted export and reporting tool. My XML language is kind of quirky, but it works quite well. To extend CFCore instead of a generic Object, set ExtendCFCore="true" in the SchemaDef attributes of your application model, and use one of the configuration cartridges that includes "msscf".
The CFCore gets extended by the MSS Code Factory itself:
I'm still running the 1.7 C#/.Net implementation that I ported from my 1.0.6 Java code base. I did a lot of clean-up porting to .Net, and more porting back to Java 6. I'm almost proud of the code now.
It's a work in progress, but a solid start.
I'm fealing really pumped. I wasn't sure I'd be able to generalize the engine, but it turned out to be fairly simple to do.
All projects now clean compile.
You'll need to reference the net-sourceforge-MSSCodeFactory-CFCore-1-8-nnnn.jar file if your project specifies ExtendCFCore="true" for its SchemaDef.
Added binding HasContainer, reference ContainerRelation, and SchemaDef.ExtendCFCore.
Some minor fixes for the copyright credits.
Copyright now defaults to the Author, not the Company. The Company only aquired copyright if no author is specified by the model.
The Web Form Prototype is now complete. This release will get posted to FreshMeat, then I'll start working on implementing the "glue" between the user interface and the application objects.
Finished debugging "poptop" directive. Used to produce subclass "matches" for prototype Filter forms.
Added the "poptop" directive. Where popto finds the first context that matches the target type, poptop finds the top-most context that matches the target type. This is useful when navigating the subclasses of a table, but you need to perform expansions within the context of that top-level table instead of the subclass.
Added iterator TableDef.PrimaryIndexColumns. Needed it for calculating HTML TABLE widths. Match list servlets now properly specify their table width.
Added a "sum iteratorName expansionBody" macro. The behavior is to evaluate the iterator, and for each iterated item, apply the remainder of the macro to obtain a numeric string value. Convert the numeric string to an Int64 and calculate the total. Final expansion of a "sum" macro is the total.
I've never needed to do this before, but in order to calculate the proper size of an HTML table I need to total up the size of the columns, which vary.
Adding the calculation verbs add, subtract, multiply, divide, and remainder. I need basic math to calculate the pixel-width of table columns based on the number of characters to be displayed for the field, and also to calculate the "larger" text edits to allow for filter criteria syntax.
First cut of the Filter forms. There's a lot of tweaking to do with them yet, but the basics are in place. Specifically, I want to double the size of string, text, etc. fields so you can see the criteria you've punched in. I'll achieve this by adding a "multiply" verb to the engine.
Finished debugging constrainMax and constrainMin.
Added the macro language expansions for constrainMax and constrainMin.
Packaged up the initial outline of the MSS Code Factory self-replication support.
Turns out there were still a number of layout issues with the View sub-pages. Those have been corrected. I can't think of anything else to do with the Views until there are more forms existing to wire buttons and menu items for.
Renamed the Java projects to incorporate the project version numbers, so that more than one version of a project can be hosted by the same JEE server.
I've opted to include a compacted version number in the Java project names so that more than one version of a project can be hosted by the same JEE server.
Finished debugging the HasLookupColumn and LookupColumn verbs. There were a lot of pieces of code to modify in the process.
The Lookup, Master, Parent, and Container relations whose "To" tables specify a LookupColumnName now produce a read-only text edit field with that field value to be displayed, and an "Open Object" icon to open the referenced object.
Made use of the new Label verb to provide a much more readable interface for most of the projects I have on the go. Still have to update 2-3 models.
Added support for the verbs HasLookupColumn and Label. RepoModelParser and XSD now support AnyDef.Label attribute.
Refactored J2EE to JEE throughout system. Fixed numerous bugs. Updated sample models.
I'm much happier with the layout of the manufactured Web reader UI prototype.
I've added a new LookupColumnName attribute to the TableDef in the application models, which is used to implement the reference verb LookupColumn within a table scope.
This build produces clean code for an outline of a Web UI prototype I'm working on for JEE Glassfish. The idea is to produce as many component fragments as possible for use by a standardized object navigation interface, which can also be reused as components of custom Web UI implementations.
There is no way to automate the creativity that goes into UI customization. 2.0 will try to help tackle the problem by providing a UI model view that can be used to generate common look and feel UI implementations with different GUI toolkits, including web interfaces. However, that's too far future for use right now, and I've got a project to implement.
Added the verb ColumnInOwnerLookupRelation for use in filtering the View attributes of an object when presented as a web form. Only data attributes which are not set by picking a Lookup, Master, Parent, or Container relation should be displayed as attributes.
Clean compile of all manufactured Java code. Good time for a snapshot.
Increased size of InitValue, NullValue, DefaultValue, and UnknownValue of StringDef to 2000 characters.
Added debugging/traceback details for certain runtime exceptions.
Corrected typos in Java BL rules for production of IBLEditObj and BLEditObj.
Increased the size of the StringDef attributes InitValue, DefaultValue, NullValue, and UnknownValue to 2000 characters.
Cleaned up a few more bugs. Spent most of my time lately modelling applications. The most important change is I remembered to re-enable the CSharp support with this checkin.
Continued working on code, nothing specific. The main reason for this update is to correct a packaging issue.
Completed the Business Logic template implementations, including the SchemaPg8BL layer. I'm no longer generating updates for the BL layers of my personal projects; their data models are sufficiently mature that I won't be seeing too many changes to them for the foreseeable future.
I've refactored the interfaces for Java and C-Sharp so their names all start with "I" rather than sometimes having the "I" midfix in the name.
There is now a Business Logic template ruleset for Java. The idea is you generate a near-final model of the project once with the "net-sourceforge-msscodefactory-generator-17-layered+bl" ruleset, and after that you maintain any new object by hand, using the "net-sourceforge-msscodefactoyr-generator-17-layered" ruleset in the future.
Java 6 and PostgreSQL JDBC layers compile clean. Completely untested, but it's a start.
Java 6 support is baaaaaaccccckkkkk!!! The ConfigurationFile implementation still needs to be converted from CSharp syntax, but the rest of the Java 6 implementation compiles cleanly for the RepoModel.
Implemented Oracle 11gR2 database creation scripts, complete with use of sequence generators and stored procedures.
Added verb HasNullableColumns for IndexDef to engine.
Now only generate sp_read_dbtablename_by_suffix() if the index in question has no nullable columns. I don't know how to do dynamic SQL in stored procedures, so those functions have to be implemented in C# or Erlang where I have control over the syntax of dynamically generated SQL.
MySQL 5.5 database creation scripts are now complete, and were executed for my test suites. Had to modify the RepoModel as it turns out that I was stepping on some reserved words with my column names.
It turns out that MySQL 5.5 stored procedures are too primitive to be useful, so I've removed them from the database creation rules.
Finished testing the SQL Server 2008R2 database creation scripts. Turns out that unlike Sybase ASE, SQL Server lets you pass TEXT parameters, allowing for a complete DbIO stored proc implementation equivalent to PostgreSQL 8.4.
Updated the database creation scripts for MySQL 5.5. Added stored proc templates based on SQL Server 2008R2. The stored procs aren't actually migrated yet, so the database creation scripts for MySQL 5.5 just perform the basics:
Increased the size of GenItem.RequiredName from 64 to 127 characters.
Database creation scripts for SQL Server 2008R2 now:
Everything including the stored procs is complete and installs to my SQL Server 2008R2 Express database. I'm having trouble with the syntax for RAISERROR, so I had to comment out some important checking in sp_update_dbtablename() and sp_delete_dbtablename() which provide last-minute detection of database collisions.
One thing I really don't like is that SQL Server doesn't support "FOR UPDATE" syntax for selects. If a stored procedure/function automatically selects for update, then this is ok. If that isn't the case, you'll still get a collision detection when the update changes 0 rows, but that's kind of a last chance way of handling the issue.
Fixed the use of dispenser defs to create ids in PostgreSQL 8.4 sp_create_dbtablename(). This was the last TODO item for PostgreSQL 8.4 stored procedures.
Fixed PostgreSQL 8.4 stored procs. I am now satisfied that sp_update_dbtablename(), sp_delete_dbtablename(), sp_create_dbtablename(), sp_read_dbtablename(), sp_read_dbtablename_all(), sp_read_dbtablename_by_suffix(), sp_next_schemaidgen(), and sp_next_tableidgen() are complete.
Discovered that PostgreSQL 8.4 does not support a "with timezone" syntax for implementing TZDate, TZTime, and TZTimestamp. None of the databases so far does. Gonna take some creative hackery to implement TZ*.
Added option "-p PORT" option to .bash scripts for PostgreSQL 8.4, which gets passed to psql as --port=PORT.
Took a new snapshot of SQL Server 2008R2 database creation scripts based on Sybase ASE 15.5 scripts (with stored procs.) Completely untested at this point.
Created PostgreSQL 8.4 stored procs from Sybase ASE 15.5 templates. Not tested at all yet.
Created Oracle 11gR2 database creation script prototypes from PostgreSQL 8.4 rules. Needs a lot of work -- doesn't even use the right command interpreter for the SQL scripts yet.
Finally got the arguments for sp_create_dbtablename straightened out to my liking. Required the addition of a couple of new iterators scoped by TableDef: GeneratedTableColumns and NotGeneratedTableColumns.
Fleshed out the initial skeletons/outlines of the remaining Sybase ASE 15.5 stored procedures. Completely untested and far from complete, but they do show the intent of the procedures.
Implemented and wired sp_next_schemaidgen and sp_next_tableidgen scripts for Sybase ASE 15.5.
Implemented CompanyDescription and ProjectDescription verbs.
Updated all the license headers to use the DefaultLicense with a description of the project and copyright attributed to the company for 2011.
I strongly encourage you to look at the net-sourceforge-msscodefactory-1-8-repo-model.xml application model for an example of how to specify a license, and apply appropriate changes to your application model.
Debugged the EmbeddedLicense implementation, fixed the LicenseDef parsing, implemented $prefixline PrefixMacro BodyMacro$ syntax. Tested all of the above.
Need to implement ProjectDescription and CompanyDescription next.
Added support for LicenseDef and reference SchemaDef.DefaultLicense attribute, with a binding verb of "EmbeddedLicense". You can now add licenses and wire them to your application models.
Next I want to add a processing verb $prefixlines prefixRuleName ...$ acting similar to the uppercase, lowercase, and mixedcase verbs. This will be used to properly embed license definitions as comment text.
Fixed the same problem with MinValue.
Fixed that nasty MaxValue bug.
First cut support for the additional XML/XSD data types for C#/PostgreSQL 8.4 DbIOs. Completely untested -- not even compiled.
I've made a fundamental change to the way the engine resolves verbs/names. When a tool is eliminated from a multi-tool toolset, that tool's rules are explicitly searched for a resolution of the name. This will allow (for example) the Sybase ASE 15.5 database creation rules to define the names of stored procedures, and then use those expansions in the C# implementation of the DbIOs.
The issues with Token, NmToken, and NmTokens implementations in C#/.Net 3.5 have been resolved.
The additional XML/XSD data types are now properly used in the $SchemaName$.xsd file.
Added Sybase ASE 15.5 skeleton created stored procedure scripts for:
Even the arguments to the stored procedures aren't right yet.
The 1.7 C# rules have been largely updated to support the additional XML/XSD data type definitions. There are still some issues with Token, NmToken, and NmTokens implementation.
All future work will be done with the 1.7 code base, and it's easy to migrate your models from 1.6 to 1.7.
Install 1.7, then use your favourite text editor to open your application model XML file, and replace the 1.0.6 and 1.6 references for the document type with: