Page 1 of 1

Solution for per client custom business logic

Posted: Sun Jul 15, 2007 3:51 pm
by Begby
Sorry if this is long winded, but I wanted to make it clear.


I am in beginning stages of rewriting a rather large app. In looking at the current code, there are a lot of custom 'hacks' within it for individual clients. Basically this system ships crap for people, and we integrate with many different customer systems, some of which are really whacked.

Example: One client sends us orders that only contain two shipment types, indicated by '1' or '2'. Our internal codes are things like UPN, MR, PM, UPS, DHLR, etc. The codes we use are pretty standard. In order for us to work with this client, we had to map '1' to UPS (UPS ground) and '2' to DHL2 (DHL 2nd day air). This was accomplished with a nasty if(clientID = X) { do mapping } statement.

One obvious solution to this would have been to tell the client to fix their system, however that is not an option if the client is worth several million a year, we have to bend to them a lot of the time. Another solution might have been a custom import adapter for the client, but there was no way for us to easily implement this and only allow only the one client access to their specialized adapter.


These customizations also occur at different spots in the system, and could occur anywhere. For instance we had to do a custom SOAP request for a client after their orders were shipped. This is also a nasty if. There are also specializations for customers who are using amazon seller central as their order system. At order fulfillment time we have to check if the client is a amazon seller then write out an XML file for each order. This amazon thing was a customization I had to put in on Friday in the old system and it sucked.

Their are a lot of problems with this in the old system. The code is a disaster that becomes worse each week, and if a client goes away it is impossible to remove all of their customizations without worry of something else exploding.


So as I see it, there are probably four different ways we have to check for custom code
1. ClientID
2. Integration platform (their shopping cart or ERP system type)
3. The client status (deleted, haven't paid their bills, active, etc.)
4. The shipment type. For instance if its USPS international we might have to print a custom form or something, or some weird freight vendor we might have to send the order to them via soap.


One thing we thought of is a trigger system. At arbitrary points in the system I want to be able to mark a trigger or event. This should be extensible so I can easily add in triggers later on without a lot of refactoring. That way if I get a crazy request like "we need to have X happen if an order is on backorder for client Y and it needs to happen right before the order is released but only if it is on a Tuesday and the order consists of products in warehouse 2".

Secondly we need to code encapsulated somehow, so if a client goes away, or an order method goes away, then the code goes away too. I was thinking we would have an override object for each client, platform type, status type, etc. In that object we would create a method with the same name as a trigger, then the trigger would execute those methods if they existed by passing information related to the trigger event, then those arguments would be passed back if they changed, like.

Code: Select all

// 232 is the clientID and I guess this class would be returned by a factory/instance map thingy
class Triggers232 extends Triggers
{
  public function OnOrderEvent( $order )
  {
     // do something custom to the order for NeedyClient
    return $order ;
  }
}

The big problem I can see with this is that we have 300 clients right now, and sometimes process up to 2000 records at once. So this would mean possibly 60,000 method calls to the triggers thing just for that one action. If this is the only way it can be done, thats ok. I can just buy more servers.

Posted: Sun Jul 15, 2007 5:38 pm
by Selkirk
I used to work on an ERP system and we had to deal with similar variation and customization between different plants that it was installed in. Our variation came from different markets (automotive vs. building materials), different machinery & product lines in the plant, and differing governmental regulations.

We did standardize all data into the same database schema. This was a mixed bag. We could end up with significant variation here, mostly based on industry. For example, winning an automotive contract might mean an order that consumed the entire capacity of a single plant for an entire model year making a single part for a single company on essentially a handful of orders. Another plant might specialize in customized products, where every order was different and where there were no specific part numbers. Each order might include specifications (3 holes, positioned here, a bevel there, etc). Creating an order entry system that handled both cases at the same time was very problematic and probably did a disservice to both plants.

Trying to fit these different lines of business into the same schema has advantages for some kinds of centralized operations. It's dismal for usability for the end user. Depending on your interaction between customers, need for centralized data, and types of variation, I wouldn't be afraid to create independent sub-systems for different customers, and then integrate them as needed for centralization. Partitioning is your friend.

Another approach we took is similar to what you are talking about with events. For example, invoices and bills of lading differ between different countries. So, we had a modular invoicing system, which we called "the sandwich." The beginning and the end of the process was standardized ("the bread"), but each plant could customize what happened in between ("the fixings"). So for example, in one plant, we might have to add an environmental impact module to print a statement on the bill of lading required by the local government. We might add modules for calculating taxes in different jurisdictions, etc. These little event handlers were the "fixings" inside the "sandwich." This system was extremely successful.

Another thing that I think was successful was that we had a rule that we absolutely never allowed comparison for the purposes of determining a course of action on anything but a type code. There were no constants in the source code except for type codes. All comparisons had to be either against another db field, or against a type code. We also had a standardized system for registering and tracking these type codes. Today, with better ORM, I'd also try to replace these type codes with objects and inheritance or delegation.

Our system had a table where we registered all of the type codes and grouped them. A couple records might look like:

id group code description
1 MEASURE_UNIT MEASURE_MM Millimeter
2 MEASURE_UNIT MEASURE_IN Inch
3 TYPE_GROUP MEASURE_UNIT Unit of Measurement

(notice that groups were also type codes)

So, a consequence of this, it was forbidden in our system to say (if customer=="Smith Industries" or if customer==935). We would have to create an attribute of the customer a type group and type codes. We actually had specific functions for this, which made violations stand out. It was also forbidden to expose these type codes to the user, only their descriptions.

So you mention having codes for shipment types like "UPS". This would not have been allowed. Our order would have had a shipper field, related to a shippers table. Each shipper would have a record with a numeric id. Then that table would have columns describing each shipper in generic terms. Each plant would then enter their shippers. They could vary the generic attributes of the shipper and thus control how the application treated them. Everything was like this.

Sometimes this did not work. In two subsystems, pricing and bill or material, the variation was so great that the system we created to handle it ended up being almost like a data driven programming language. These systems were successful in that they encompased all of the required variation and worked. However, they were so abstract and so configurable that only a few people could learn how to use them. I think we might have been better off here partitioning into 4-5 different pricing systems and 4-5 different BOM systems and then integrating them.

We ended up with about 500 or so tables in the system. Overall the system was extremely successful with a few drawbacks. It first went live in 1994 and is still in use today.

I was responsible for both the design of the invoicing sandwich :) and the pricing system :(.

Posted: Sun Jul 15, 2007 8:18 pm
by Begby
Thanks for the input!

Luckily I think the data is going to stay consistent for this system. The variation will be in how we take in orders, and how we ultimately output those orders, but in the database they will always just be orders. Luckily it doesn't have the complexity of a full blown ERP system.

Perhaps I should start looking at ERP design though to get some more ideas. I got the idea of the triggers/events from a UPS system that I will be integrating with that offers a lot of customizeability through event driven VBA scripting.

As for the ability to completely replace subsystems, that is a really good idea. Along with the event driven stuff, perhaps I will think about using factories to return objects such as input adapters and such. Lets say a client is sending us an order via soap. Instead of just getting an instance of the XML adapter, I would instead call a factory and pass it a client object.

The client object would contain the client ID, their integration type, and other info. The factory would check to see if a custom adapter existed for that client or integration method and if not return the vanilla XML adapter. Another way to do it would be to store within the client record information about specialized objects, that way the factory wouldn't need to check to see if a file or class existed.


That is also good advice on the type codes. The shipment types are stored in a table, but unfortunately throughout the old source there are comparisons against strings. This has created some bad problems that we are currently dealing with. It is so hacked that we are handling orders shipped before a certain date against one set of rules and newer orders against a new set of rules.

Ultimately though I would like to eliminate arbitrary comparisons if possible. If a different course of action was needed, like for a specific client, I would rather have that code somewhere else so I would not need to modify the core. This would keep it clean as customers come and go, as we gain new warehouses, and how shippers change their minds about stuff.