![](\"https://lh5.googleusercontent.com/v1PbZaRZQJjKJbBQiQZOXRUGjLK_c000j3edWpNb7jhim2dDHl21ircQVV4-aabJnHA57-A68J7WXfTkMFFxeVro0kGAlSMh2PKZ72yW5IezNvN-FiM4Xm-siC7PhAS8ZarnG4imewzKJhjKxAR7Mt0A3R5BOkpSCDYW14ldqG2CLpW5XC-jbwJtvg\")
As you may have read about Application Cockpit in one of our previous articles, it is “The tool which lets concerned developers quickly find their bearings and gather relevant information pertaining to a particular service running in our ecosystem”.
\nOver time, AUTO1 engineers and managers grew fond of using Application Cockpit, and started relying on it on a daily basis. The increase in popularity, naturally, led to development of new features, out of which, one of the arguably most impactful ones was the \"dynamic service search\" (or \"App Cockpit Search\", as we call it internally).
\nAUTO1 platform currently consists of 600+ microservices, and on average two new services are created every week. Many times in the past, we needed to search our AUTO1 service inventory for services which were matching some specific criteria, e.g. are written in a specific language, using certain frameworks or libraries, or have certain features enabled.
\nAs it happens, this is actually where App Cockpit App Cockpit Search chimes in. From the user’s perspective, it is an UI that lets the user input some search criteria using the Cockpit Query Language (or CQL for short) and be presented with a list of services matching the criteria. One typical use case is searching the AUTO1 service inventory for the usage of a specific library in a specific version.
\nBelow is an example search for services using org.springframework.boot:spring-boot-starter dependency in version 2.3.0 onwards:
Once we were certain of what we wanted to achieve, i.e.: search our AUTO1 service inventory using dynamically defined criteria, we began the efforts of defining the architecture.
\nAs you may recall from our previous article about the Application Cockpit, we use Backstage as our Front-end. Backstage is fetching data from our backend-for-frontend service called “Application Cockpit Data Provider”, which in turn, is backed by a PostgreSQL DB.
\nAs Backstage itself comes with search functionality that could be potentially used for our purposes, we were left with two options, either to reuse the already provided solution and tailor it to our needs or create a solution from scratch.
\nAfter evaluating the Backstage search, which at that time, was still in the early alpha stage, it was clear to us that it would be a bumpy ride. As it is a full-blown search mechanism, or to put it more bluntly, an over-blown mechanism, compared to what we really needed:
\nAs we already use Spring Data JPA as the persistence layer to communicate with PostgreSQL in our data provider service, we have limited our options to two choices: dynamically built queries on top of joined database tables or database views. Either of them seemed to us, initially, as a good candidate for dealing with the complexity that stems from the heterogeneous data structures.
\nThe idea behind this solution was to create a set of SQL queries, backed by corresponding Spring Data JPA repositories, one for each attribute in question, execute these queries against the DB, and then combine the results on the Java side performing the necessary filtering and aggregation.
\nThe advantage of this solution was that we would keep things small and isolated. Each query is in a separate repository. Each one would deal just with the data structures it's aware of. On the other hand, this solution would come with a significant performance penalty, due to the substantial amount of data that would be transferred from the DB, just to be later filtered out on the application side.
\nThe alternative solution would be to dynamically build a large SQL query that would perform both the retrieval of the data and the filtering and aggregation of its results.
\nHowever, this approach would come at an increased maintenance cost. As every data structure change would require a corresponding Java code change, hence we would lose the benefits of having the data values stored in the JSON objects. On top of that, maintaining a large SQL query would quickly become a problem on its own.
\nKnowing the shortcomings of the dynamically built queries, we decided to evaluate another promising approach - the database views, that could hide the complexity of dealing with the heterogeneous data structures:
\nIn that approach, we would create a database view that could look like the following (based on our previous example):
\nwhich returns:
\nThe main database view, here called document_view, would be the aggregation of multiple sub-views. Each sub-view would only know about its data structures, hence we would achieve the desired single responsibility and encapsulation traits, making the solution more extendible and more maintainable. Even if something would be changed within one of the sub-views, it would not affect the aggregated view or other sub-views.
As previously mentioned, since we use Spring Data JPA extensively, the database view would be reflected in the code as a JPA entity which would allow us also to use the Criteria API to build a generic query on top of that main view and utilize the built-in PostgreSQL functions, that would come especially handy when interacting with the JSON objects.
\nAs the DB views seemed like a good solution, we still needed to answer one more question: whether to materialize them or not. As materialized views would be, obviously, faster to query, but that would not come for free, as we would need to build a mechanism to frequently refresh the materialized views, which would also induce a performance overhead. In the end, as we expect the App Cockpit Search to happen fairly infrequently and we value accuracy over query execution time, we’ve decided to not materialize the views.
\nFinally, we decided to move forward with the DB view based solution. Although it’s not a silver bullet and requires us to keep an eye on the query performance, as over time the number of sub-views will grow, which might impact the overall search performance, we consider this to be a fair trade-off for the gained extensibility and maintainability of the search functionality.
","fields":{"slug":"/harmonizing-application-cockpit-data-using-db-views/","tags":["Cockpit","PostgreSQL"]}}},{"node":{"id":"cfee7911-e765-5a6e-ba90-1eb2ce96d837","frontmatter":{"category":"Architecture","title":"Application Cockpit Model Outline","date":"2022-10-28","summary":"A short trip around the AUTO1 Application Cockpit data model","thumbnail":null,"authorName":"Przemysław Walat","authorDescription":"Przemysław is a Expert Software Engineer based in our Szczecin office","authorAvatar":{"relativePath":"pages/application-cockpit-model-outline/avatar_Walat.jpeg","childImageSharp":{"resolutions":{"base64":"data:image/jpeg;base64,/9j/2wBDABALDA4MChAODQ4SERATGCgaGBYWGDEjJR0oOjM9PDkzODdASFxOQERXRTc4UG1RV19iZ2hnPk1xeXBkeFxlZ2P/2wBDARESEhgVGC8aGi9jQjhCY2NjY2NjY2NjY2NjY2NjY2NjY2NjY2NjY2NjY2NjY2NjY2NjY2NjY2NjY2NjY2NjY2P/wgARCAAUABQDASIAAhEBAxEB/8QAGAABAQEBAQAAAAAAAAAAAAAAAAQDAQX/xAAWAQEBAQAAAAAAAAAAAAAAAAABAAP/2gAMAwEAAhADEAAAAYNcbctZHqE5AEDf/8QAGRAAAwEBAQAAAAAAAAAAAAAAAQIDEhEA/9oACAEBAAEFAnyRJdo0aBmiW9jE1khCKBOxPVY8/8QAGBEAAgMAAAAAAAAAAAAAAAAAABEBECH/2gAIAQMBAT8Beji//8QAFBEBAAAAAAAAAAAAAAAAAAAAIP/aAAgBAgEBPwEf/8QAHRAAAgIBBQAAAAAAAAAAAAAAAAERQQIgISIxgf/aAAgBAQAGPwJvpqi/CMJg4KTFN7l6P//EABsQAAMBAAMBAAAAAAAAAAAAAAABESExQWFR/9oACAEBAAE/IYU8XDGNg3v6Gar2x1WU9pKgnlpCG63tFjubo1NwXmn/2gAMAwEAAgADAAAAENfwvP/EABgRAAMBAQAAAAAAAAAAAAAAAAABEUFx/9oACAEDAQE/EHENHQ0h/8QAGREBAAIDAAAAAAAAAAAAAAAAAQAhEBFh/9oACAECAQE/EANUy+Z//8QAHRABAAIDAAMBAAAAAAAAAAAAAQARMUFRIWFxgf/aAAgBAQABPxB5rcKC843EOZKEWYB/dSvRVSN3XnMCCmUgo98hrV7b0d+xivco2sPKSmmz7hteDByVo6Gf/9k=","width":50,"height":50,"src":"/static/e4e77cd92050dbb448a088991f2d0c50/d2d31/avatar_Walat.jpeg","srcSet":"/static/e4e77cd92050dbb448a088991f2d0c50/d2d31/avatar_Walat.jpeg 1x,\n/static/e4e77cd92050dbb448a088991f2d0c50/0b804/avatar_Walat.jpeg 1.5x,\n/static/e4e77cd92050dbb448a088991f2d0c50/753c3/avatar_Walat.jpeg 2x,\n/static/e4e77cd92050dbb448a088991f2d0c50/31ca8/avatar_Walat.jpeg 3x"}}},"headerImage":null},"html":"Welcome, fellow concerned developers. In our previous installment, we have promised to delve deeper into the model that Application Cockpit is using underneath to see how it synergizes with the high-level architecture in order to make the whole product simple yet powerful and extensible. Be advised, that while we have always aimed for simplicity, our initial take on the model was a little bit more complicated, but after more-than-a-few heated debacles, we have stripped it down even more and so far we are pretty happy with it, despite more than a few \"I wish we had the initial design in place\" moments of weakness.
\nIn honor of the early version, we are also going to take a short trip down the memory lane to see the trade-offs and compromises we have decided to make along the way and features we have decided to add. We are going to discuss the direction we took and what were our reasons to do so (or in other words, \"what were they thinking ?\"). But for now...
\nAs we hinted in the previous blog post, Application Cockpit is actually binary, in a sense that its primary domain consists of only two data types: Resources and Readings and the (extremely) short story is that a Resource is an entity for which we are going to track Readings, while a Reading is a piece of information pertaining to a given Resource. The longer story follows below.
\nLike mentioned above, a Resource is mostly something that we are going to collect readings for, it can be a physical entity, or a purely logical one. Examples of currently tracked Resources include:
\n(semi)physical
\nLambdaDeployment
\nRdsInstance
\nServiceDeployment
\nServiceInstance
\nlogical
\nLambda
\nService
\nSoftwareDependency
\nSoftwareDependencyVersion
\nThe Resource has a very rigid model, comprising of only a few, well-defined properties, that is:
\nid - the unique UUID
\ntype - the type of the resource, see above for examples
\nname - a textual identifier, unique across one type; needs to be deterministically derived from resource's properties available at runtime (further details in tradeoffs chapter)
\nparent_id - id of the parent resource (if any)
\ncreated_on - a timestamp and a late-comer to the party, but we find it useful in many ways, especially during troubleshooting
\nIn order to publish (a.k.a create) a Resource, the collector will provide only the name, type and optionally the parent identifier.
\nWhen it comes to type, it is a free-form String. We have no predefined enumeration to represent it and the collectors (see previous article to learn more about them) are free to push whatever types they want, akin to the front-end which is allowed to ask for any type it wishes.
\nIn addition, the inclusion of parent id allows us to build hierarchies, so that our data closely follows the domain that we are modeling.
\nGiven the above, you should now have a general idea of what a Resource is and how we use it, so we can move on to Readings. Since we have established that it is generally a piece of data pertaining to a Resource, let's have a look at its properties:
\nid - the serial identifier;
\ntype - type of the Reading; free form String, akin to Resource type
\nresource_id - uuid of the Resource this Reading describes
\nvalue - free-form json document to hold the data of the Reading
\nstoragemode - determines the behavior when multiple Readings are published with same resourceid and type:
\nSNAPSHOT - only the most recent Reading is preserved
\nTIME_SERIES - all of the published Readings are stored
\nSince the values we store for Reading are free-form JSON, it gives developers a lot of flexibility in how to handle their flows. We store both trivial things, e.g.:
\nTo the humongous too-big-to-fit-in-the-picture guys:
\nAgain, akin to Resource type, the Reading's type and value are completely opaque to most of the Application Cockpit infrastructure, so individual contributors are free to ship anything they wish in those properties. The only components which need to be aware of the exact structure are the Collector that is originally publishing the data and the UI component which is querying the Data Provider in order to render the respective view. That way, many developers are free to add their own custom extensions, without stepping on each other's toes or needing to seek approval or assistance from the core Application Cockpit team, which in addition can remain blissfully unaware of the atrocities that the others may commit and attempt to hide within their custom Readings.
\nHaving dealt with the detailed model description, we can now move on to discuss how our journey with its design looked like. Of course we, being the pragmatic, more-than-once-bitten little members of the software engineering club that we are, were under no delusions about what is going to happen to our nifty design when it gets into its first bar fight against the necessities of real life. The only question was when we are going to be faced with our first moral/design dilemma and how much are we going to be willing to compromise our collective conscience.
\nWhen it comes to the little trade-offs, actually, the more observant reader might have noticed by now that something is a bit amiss in our picture:
\nIn this article we have learned that in order to publish a Reading for a Resource, the Collector probably needs to provide the resource_id to bind it to the right entity
\nIn the previous article we have mentioned, that Collectors are generally stateless and have no access to the storage holding Resource data
\nIn consequence, how can a Collector find out the correct id to publish the reading for ?
\nWell, it would that it could, but it can’t, so it shan’t. It’s quite a bind we find ourselves in, right ? We were adamant about not allowing collectors access to our DB, so we had to find another way to make things happen… A short look at the ReadingPublishMessage model, might provide some hints, though, so let’s start with that:
\nDoes anything stand out to you ? It is almost a reflection of a Reading, but with a slight twist, and you probably got it right, so enter Resource's name - that tiny little property looks like it has been put there in order to provide a human-friendly name for a Resource, but it couldn't be further from the truth than that. In fact, initially it was never there (we use Readings to store human-friendly names of the Resources anyway) and we added it specifically to solve the issue at hand.
\nIf you recall, the description says that it needs to be unique and possible to deterministically assemble from the data available in the runtime. Leveraging that, the Collector doesn't need to know the right id, it just needs to provide the correct textual name. The good news is that we have a way of overcoming the initial obstacle, but there is also bad news... This forces every collector to be aware of the logic required to construct names of the Resources it is interested in, moreover, the logic needs to be in multiple places and it needs to be synchronized. But let's leave it at that, since data ingestion is going to be the focal point of the next article in series.
\nAt this point a meticulous reader with a penchant for domain modeling might begin to wonder:
\n\"All right, a Resource can have a parent... That's nice, but is that really all that there is? You say you want to closely match the domain, but not everything is a child of something, sometimes you might want a Resource to 'own' Resources, so... how about that, smarty pants ?\"
\nThis was also something we were intensively brainstorming about and lots and lots of coffee cups later (and probably more than a few beer mugs, too) we have arrived at something we were not very uncomfortable with. Our main priority was to avoid complicating the model (remember ? we want people to integrate easily, so we want to keep things simple), so we didn't want to introduce any new concepts or tables. Instead, we elected to cheat a little, so please have a look at the example below with a picture being worth a thousand words:
\n![](\"https://lh3.googleusercontent.com/9ofjHjm_CHNEKWH-7C6kdE5aq9ELbeH4ekpOoOZcDMJtwYjP3xWOTiceE3kW1i0pl4H3uTCbh0c5eY1MMqpWJRXjB0J4L66x0vf6YHhs15fY-s_oA-8qobF_eynejn8xlsns09OlmgIGAy7bZNDDebFodPEHmGrPgp8HnAhoCCUIzEi7fV1OWheP\")
Basically we have elected to model a \"has a\" relationship between two Resources using an intermediary Reading to store ids of the referenced (one or many) Resources. In our example a Service Deployment has a Reading Called \"Service Deployment Dependencies\". The main purpose of this Reading is to hold data which we can use to construct id of \"Software Dependency Version\" Resources that our \"Service Deployment\" has. As you can see, if we wanted to obtain vulnerabilities of all dependencies a specific Service Deployment Resource has, we could easily construct ids of each and every \"Software Dependency Version\" and then query for its \"Dependency Version Vulnerability\" Readings.
\nThere aren't any places where we use this not-so-orthodox approach, but there are some and the example below is the most prominent real use-case. We are closely monitoring the performance impact it has on our service and so far, it has been negligible, or at least negligible enough that we are still able to look at ourselves in the mirror.
\nHaving confessed our most prominent sins, let's wrap things up with a less controversial story. There are several items that were not part of the initial design, which we have decided to add along the way, because we have found them useful.
\nThe storage mode for a Reading was a last-minute addition to the picture, but it serves its purpose well. Initially we thought of Readings solely as means of representing the current state of things, e.g. the single-point-in-time data like:
\ntech stack given service is using
\nAWS account id of the deployment
\nIP address of an instance, etc.
\nThis meant that we were only interested in the most recent value of a specific Reading Type for a Resource, always replacing it whenever a new value was collected. From the more technical standpoint - we could get away with performing upserts on Readings with the key being comprised of resourceid and readingtype. However, after a few initial releases, we've discovered that there are certain pieces of data more linear in their nature, which Application Cockpit users could really appreciate on a daily basis.
\nA prominent example of such an item is Service Events Graph, which as you can probably guess represents a service's life cycle timeline, so a mixed bag of Service Deployments, Instance Startups and Shutdowns, Incidents etc. Here's what it currently looks like:
\nWhat we have also found out rather quickly was that it would be quite awkward to represent this kind of data using our initial point-in-time-based approach to Readings.
\nEnter storage mode - a property of a Reading which tells the underlying infrastructure how to behave when a new Reading for the same resourceid and readingtype is persisted.
\nTo preserve all the already existing flows, we have assumed the default mode to be SNAPSHOT, which upholds the default behavior of upserting new values, keeping all the implemented features intact without any additional effort.
\nAs an extension of the functionality, we have introduced TIMESERIES mode, which preserves all the values that were collected and allows us to construct e.g. neat timelines like what you can see in the screenshot above.
\nAs a bonus, here's how the actual data underlying the graph looks like:
\nWhilst it seems quite natural for a Reading to have a timestamp since it is collected (or, as you will, measured) at a specific point in time, it's not so straightforward in case of a Resource, especially not from the Business Domain side of things, since it mainly represents a certain Entity that simply is. Seems like we're not all about Business, I guess, but we'll get to that in a while.
\nWith the above in mind, the first iterations of the Resource model did not have a good reason to include timestamps, so they didn't. However, as we kept adding new features and collectors, we have started noticing that the system is accreting quite a lot of Resources and that a rather large portion of them is only relevant within a short timeframe. It wasn't a problem for us at that point, but even though we hate doing the house chores as much as we do, why not keep our room tidy and \"let go of the things that don't matter anymore\", as we've all certainly been told at least a few times in our life ?
\nThe part where it most stood out for us were Service Instance resources which by their nature are short lived, since as soon as the instance is dead, we don't care about it anymore (apart from audit reasons perhaps). The point was - how do we know whether we can safely delete the Instance Resource ?
\nAs you have seen above, we have ServiceEvents telling us when instances were terminated, but can we rely on the event to always be generated and are we willing to trust it ? Not really.
\nAlternatively, we do track instance health checks but they are not ideal by themselves, particularly since a freshly minted Instance would have no checks at all in the beginning and then report as unhealthy for some time before it becomes stable, not to mention that in some cases it is all right for an instance to fail some of the inspections for a while and still survive. The list of things to consider goes on and on, which went contrary to our goal which was to avoid creating a too-sophisticated cleanup-logic for fear it would become too brittle to be effective.
\nThe introduction of Resource timestamps allowed us to reach all the objectives in a relatively low-cost and simple manner. Once we started tracking when an Instance was created it was a simple manner scanning Instance Resources on a schedule, discovering those which had no health reports for quite a while and were not created recently to safely trim them away from our storage.
\nThe added bonus is that having a Resource creation timestamp comes in handy during investigation of issues (not that there are too many), since it makes it easier to track the timeline and wrap your head around what happened when and why.
\nAt this point I am going to conclude my musings and I hope that you've found them interesting or perhaps even inspiring at some point. The topic is far from exhausted, especially given that our journey with Application Cockpit is still well under way and we keep on adding new features into the application, so please look forward to the next article in series.
","fields":{"slug":"/application-cockpit-model-outline/","tags":["auto1","Cockpit","DataModel","Architecture","Core"]}}},{"node":{"id":"4c23b9b2-0718-52ff-a466-8b87fec6925d","frontmatter":{"category":"Architecture","title":"Machine Learning pipeline","date":"2022-08-31","summary":"Machine learning systems are complicated, but good infrastructure can reduce this complexity by automating processes. Well thought out infrastructure solutions can lower the engineering time and accelerate your teams ability to deliver ML applications. In this blog post, I go over an example of automating a ML pipeline using AWS services, review the structure and give you some action points to consider when creating your own.","thumbnail":null,"authorName":"Alena Nazarava","authorDescription":"Alena Nazarava is a Data Scientist working remotely from Bavaria","authorAvatar":{"relativePath":"pages/machine-learning-pipeline/avatar_Nazarava.png","childImageSharp":{"resolutions":{"base64":"data:image/png;base64,iVBORw0KGgoAAAANSUhEUgAAABQAAAAUCAIAAAAC64paAAAACXBIWXMAAAsSAAALEgHS3X78AAAEgUlEQVQ4yyXRaWwUZRgH8EnEkxTXcrVFzpJSIUBtqdB272Nm53xn3pnZOXZmdmZn9u6W0m5blkKtUIkCtQiklOOL1MolEEBDBQP1SEwUNcaEeEXRT4qJH4x+aiJuNXnyfPs9/+T/ILIiSBKURSDxpASJGMRFFhfYqERjMkAVJsJhHoHwiyDMM0ERhHhQ2WEFYCrAkTks8zGRE3lK4HABRHkGjzFRlSF41P9cjatu/rz6RQtI/zaVQ2U6HGMiMojIEFO4/7EEeQFwHAUZjKMikEJFMqzQGLqpvhxp1FqWzUeQtbULFRBR6YgydzeqcngckoiiirEYFDiGAyRHoSwRAtEARANoW/OIFng4c2D27MB+6Kt5cp4Q9WpsRKZCCo3GWVyH1H9Y4gTIQICzJArQIAi6uZDH29R4plf+/mjXr+OF6e1861IXCLXpHKbQIQ2gOoslBApR4xUMBZ6GLMESERDxg0AHG+7wNa17e8j5c3L3g/HinZLqrnMxnhaDxys+AaOmSFoSU8ExReZlAYgczVEYQH1MyMMF2wMtGw4n8L+m9j4YzV028c1Vj9DuzRVswGgyRtky4ygA0XRFU2OqxEkQCDQBsSAI+dmQh2xvZpvX/jJ14IcRm1heve4JhGxdbwpEZWyJcWQmpbGIXsG6rKliXGQljhYoDEb8pN9NtG9pXVH75cnShURHU9W8zQse9TSssGU6GaMdicmoXNaASCKhGUbc0BRdEeIiUDiSRf39cjQDqRFHnL09OtOLMfWuTdWPN7iqFCacUumUwmR1Lm8KSNI2kqZm6IqhyYYCFUBIhP/GWM/0+clPbt+cPvnauZGSZ9XCjYuekhuWHoz7KuF5XchbQtGWENu2kpZhJuIVr8ucwuKVtq9NvHTvq7vnLr5z5eLV0f7iOtdjgZXPnNHd34zqOYkqJKWiI3WnVcRxbNs2rblwWeVpmSFxd9urw8Off3r3i49nThw9zW5t6lhTJzYuvJbx3j+eLPDh7Sm1Jx3vzWpIOuXYjmVaWqVzGQI+irGhwJtvTL5/68OP7sz0pbM6gUc3rmFWPf1u1nP/dbUoBLszWiln9BUsJJV2ko5pWYmErqoCB6MYj4Y/++DWW5fem5w4PrGrc0cm317r8tdUndWe/208WTZBVyY+0JkodztIJp/J5jLptGMauirwkMRJb8fLI4cuX705NT42NXGEcnt9y6o9tVU3iv7ZSwP78rEdebPcZe8pZZF8MV/ozKVStlGpW2AhTUTat3V3dv/843dnL1w9eOgYumG9/9nFVOPy+8esf64MHOoxejqtod7s0M4ikitk09mUnbR0XZV4AAgMD3hZX3j/zlcOj57aUxp0r65vq1sS9239/Xzfw+uDY/1WqWgP93cODXYhc7HZVNKxdEMRIQA4GnRvdWQ9K+TQTQH36uYtNWs2Vi+Wg94/rgzNXt91sJQYKKZfHOgeHuxF0l2FVCZlOmbcUCHPUATqfqF1b3n3va+/PX3qwokjlVf5W5asDKxd/dNk6e/p4X09dnlHYVdf155y6V+to7XKa9wQBAAAAABJRU5ErkJggg==","width":50,"height":50,"src":"/static/9ff0aff34e80697669a1119f76326bbe/45876/avatar_Nazarava.png","srcSet":"/static/9ff0aff34e80697669a1119f76326bbe/45876/avatar_Nazarava.png 1x,\n/static/9ff0aff34e80697669a1119f76326bbe/eb85b/avatar_Nazarava.png 1.5x,\n/static/9ff0aff34e80697669a1119f76326bbe/4f71c/avatar_Nazarava.png 2x,\n/static/9ff0aff34e80697669a1119f76326bbe/9ec3e/avatar_Nazarava.png 3x"}}},"headerImage":null},"html":"Developing a machine learning pipeline generally requires a custom approach. There is no off the shelf, well-architectured system that will work for all use cases. On the contrary each use case requires you to design the structure that will satisfy your particular requirements. The process for developing, deploying, and continuously improving them is quite complex. There are three main axes that are subject to change: the code itself, the model, and the data. Their behavior can be hard to predict, hard to test, explain and improve.
\nThere are some aspects that are important to know before you start building the system. For example:
\n\n
For this example, we will take the use case of making vehicle recommendations for our Newsletter. The newsletter is sent out once per day and will contain a set of recommended vehicles. As additional information, we have a lot of new items added to our inventory each day. Therefore our system will have better performance if we train the model more often. Since the newsletter is daily, we can make a single prediction and have the data ready for the newsletter to use.
\nAnd of course, rather than perform these steps each day, it makes sense to automate this process and save our time for more exciting challenges. With this in mind, we can start answering some questions and determine which components we need and what the general structure of the pipeline will be.
\n\n
Even with a described use case and defined requirements, there can be many possibilities for design structures. We are extensively using AWS as a cloud provider. So, we will use SageMaker to automate training our model, and as storage for the temporary results, an S3 Bucket. Data preprocessing is a separate step and will be implemented outside of SageMaker. The final consideration is how we will deliver the predictions to other teams.
\nBy default SF sends the output of a previous state as the input of the following state. It provides service integration, which means it can connect and use other AWS services as steps. During execution there is an available context object, which contains info about the state machine and execution. This allows the workflows access to information about a specific execution. There is also an execution log, where you can find info about tasks and time spent in each step. This can be helpful for debugging or tuning your execution. This is how all these services were brought together to implement the pipeline using AWS services. We are extracting data from the Data Warehouse to Glue jobs where the preprocessing happens. It then stores intermediate results (as a least effort solution) to S3. The Training job starts as the next step after the Glue job finishes. Keep in mind that the name should be unique within the region and account and it is a required field. In the Step Functions we also specify the S3 location for the output artifacts. The next step is to set up a schedule. For this purpose we created a Cloudwatch event that triggers a state machine and executes the pipeline. The last step in the pipeline is to load output data from S3 to a database that will be accessible through an API. This task is solved using Lambda. It is another AWS service that runs code based on the incoming request or event. In our case Lambda populates predictions from S3 to Redis. Redis is an in-memory structure store that not only supports a caching solution but also has the ability to function as a non-relational database. It will override any specified fields that already exist in the hash, or if it’s new, create it. It works well for our use case. Finally, we created an endpoint using FastAPI that fetches data from Redis and provides it to the other teams. To create all these services manually and deploy them would be time consuming and difficult. Instead we are using Terraform. It is software that creates infrastructure and allows you provision resources quickly and consistently and manage them throughout their lifecycle. By treating infrastructure as code we can avoid costly mistakes that can come from manual setup of services. Here is an example of Terraform instructions that manages schedules in CloudWatch and triggers a Step Functions. And this was exactly the case, that fitted well into SPA paradigm. SPA stands for Single-Page Applications - an architectural paradigm, that divides elements of the page into static and dynamic. Static elements of the page (such as page layout, navigational menu, user information) should be loaded into client's browser only once per user session. The page layout rarely changes. On the contrary, dynamic components of the page can be updated continuously. Regardless of the technologies involved for rendering a page in client's browser, main difference between SPA and non-SPA approaches is whether a browser First picture represents traditional request where full page contents are passed from web-server to a browser on each request. Second picture illustrates SPA approach: full page content is fetched only once, afterwords only dynamic content is downloaded and full page reload does not happen. Starting from a monolith application, where HTML was rendered on a server-side, we gradually moved to our current state, where a separate application serves SPA for us. But let's go step by step. Adopting of new front-end technologies, such as JavaScript library React, allowed us to start building first SPA pages. React code is executed in the web-browser. Front-end (React) developers became responsible for all the visual representation of a web-page. For organic integration with new technology we updated existing workflow of generating HTML of the page. Specifically, we introduced several new endpoints, that provided essential information for rendering HTML page layout. Those endpoints were: Front-end developers wrote React code, which could fetch data from aforementioned endpoints and render HTML5-compliant page layout. Now we needed a way to pass this React code from web-server to client's browser during initial page load! This was achieved by introducing of what we call SPA-controller: set of php application's code (written with Symfony framework), which returned very basic standard html template that contained latest version of the React code. Being executed in the browser, this code rendered HTML contents of specific SPA page. (if template was not cached, or TTL of the cache is over) do a request to CDN, that contains actual version of front-end assets The response contains lightweight HTML similar to this: As You can see, SPA index page contains none of the data that should be shown on a page. This data will come as a result of subsequent requests.\nStages of building SPA page are illustrated in the diagram (notice 3 phases): As a result, we implemented a Golang application, that did exactly the same, as it's predecessor-monolith, but outside of php environment. The current page lifecycle started to look like this: This is pretty good, huh? Of course, there are challenges in current approach also. We are investigating the best way to deal with user locales in SPA index, we learn how to do proper caching and authenticating user, but this is a topic of another story! The AUTO1 story started in 2012. In these six years company grew a lot. From one website\nwritten in PHP it evolved into a huge and complex system consisting of large number of components.\nThis article describes the evolution of AUTO1’s architecture from a monolithic approach to Microservices. The brand independent automotive company owns business units like auto1.com, wirkaufendeinauto.de and autohero.com. At the beginning we build several classic monolith applications that served different parts of business. The first phase was a great beginning of our journey to the world of microservices.\nWe got amazing results and most of our problems were solved: Unfortunately there is no silver bullet. The microservice approach approach has its drawbacks and we faced some of them. To resolve last of this problems we introduced the SPA approach.\nWe started using React. It allowed us to call microservices directly from the frontend with proxying requests through our monolith applications.\nTo be able to do so, we introduced simple proxy layers just for communicating with services.\nMore about this approach you can read in our blog article. As a result our architecture took it’s next look: For the last six years we passed a long way from old fashioned php websites to huge distributed net of microservices.\nHere are some numbers at the end of 2018: Microservices brought us lots of advantages, but along with advantages new challenges had come.
\n
\nSPA paradigm
\nrequests each time full page content (including static content, such as layout, menus, etc), or it requests only the important (dynamic) content. This difference can be seen in a diagram below.SPA evolution in AUTO1
\nFrom Monolith to php-rendered SPA
\n\n
\nUnder the hood of SPA controller we implemented such functionality:\n
\n\n
\n\n<!doctype html>\n<html>\n <head>\n <meta charset=\"utf-8\"><meta http-equiv=\"x-ua-compatible\" content=\"ie=edge\">\n <meta name=\"viewport\" content=\"width=device-width,initial-scale=1,shrink-to-fit=no\">\n <title>SPA index</title>\n <link href=\"https://cdn.xyz.de/1.2.3/main.css\" rel=\"stylesheet\">\n </head>\n <body>\n <noscript>You need to enable JavaScript to run this app.</noscript>\n <div id=\"app\"></div>\n <script type=\"text/javascript\" src=\"https://cdn.xyz.de/1.2.3/main.js\"></script>\n </body>\n</html>AUTO1 Group
\n#spa #monolith #mvc #engineering
","fields":{"slug":"/spa-evolution/","tags":["auto1","SPA","monolith","MVC"]}}},{"node":{"id":"3e19b9f8-fd63-5270-9ae1-29f538ecd8a9","frontmatter":{"category":"Architecture","title":"Migration to Microservice architecture","date":"2019-02-12","summary":null,"thumbnail":null,"authorName":"Alexander Egurtsov","authorDescription":"Team Lead Software Engineer","authorAvatar":{"relativePath":"pages/migration-to-microservices/avatar-big.png","childImageSharp":{"resolutions":{"base64":"data:image/png;base64,iVBORw0KGgoAAAANSUhEUgAAABQAAAAUCAIAAAAC64paAAAACXBIWXMAAAsSAAALEgHS3X78AAAD5UlEQVQ4y0WSy28bVRTG5z9gi8SSv4AFCySEkNggFqgVi4K6QSoqqgQBsUkfIJUFFKqKJiWgWrRNG/pQHk3TlpTUcgKOg+14HHvssed5Pe/XnRm/YztjxzFnbBWOPl2dxfzO9917htA0S9dtQ3f+k65ZluU7uK7IWkWQTBM7TlXgJY5F5RJfKDD5HE1m8qkkSZiOZzm+hUHV8LQ9168jXlyY+WH+4tQXH7534vj7d+YXTMMVBRlGsIxYojmKorPZPIH9uuc3vCoImppfbRimPXvus1PvvDb9wdvTx996/ZWX3nj15cjcHMY1JMowYhyBKxRKRL3RAjWaoWq1xn6vn46tXznz0fz5zzfnvkMbi2b8qfB4/upXn0gVRVMtSdKQqEx4otXutPe7EzWa7f5wtBSZvXz6RCcTG/n6UbcBGtnC+k8X6AKFsa8quiLrCMk8LxKd7kG3F4Cgabc7o9Fo5f69T999k7xxZe9+xE/H9M01Zimy/OM5UUSuWzUMS9NMWVYRkoiDoB+MBU2vd3A0GlHxv2bOnNyavRj/5bK+dtuOrVH3fmXXH/iuj13ftrFp2ppmSJJKDF5UfzAIgmAwHDZ8/9rUKeX54kgtC4uRQNgbOaKWitWabXhRx4WNYPBXVZ04HA5DjWswOAyCA0hOJuKPLk17WytPvvmysnyzvvPY5grwmNXxRhzsWZaj6yZxdHQ0fFFjfgCz9tutmbNTOz9/Ld262tp8yD6K2KbR7vRq9aZfrbte1XZc2GgIT2rCAwy971W/nzr97cfHCkvXiyvXEw9/b3WCVnsf1gH8xNy0nP/hCQ/mEDubE54sbVy7MH3+5LHlS2ejt39TVSvo90O40QJzDOY2JuDTMRbC0ENm264lU2w0RidTQvmf5IGQTSw/2N1lm63JH9Gp1RuT5CE8KQhcrbVRxS6VVIY1sntoJ8mld1GqKN64u55O8WQW7eUqsuwADMlDeGLYaHZ4wSozRqms84LJ8SYnWILosIzxLEaRlMpxVp5SMiRK7LCCaAIP1w6d9zu97SSbKyglziyxBsdbQkjaAKMKpouKJLmi6Aq8AyOSST6dEVyvht0xnKXE1ae7ZF6iGb3I6GXOgBQiAtiuVFyIA+Nk2asgzLEWhP97u1yRTNfzicHh4eofqWfR/F5BylKVXEGmWa1YViG5IITmJcZACMMUUQzPYlHbTjAcb4Rwt9tbXN2OxWmAU6QQ3cwXSwrNaCVGZTiD5U2Wt3mIIHkiwuWywbJWYocrFhUHu4Ru4JsLzzfjdCaHolvUn9HsnbsbaZILR7AaAzBnwuOLyAH/3YyYyYhbW0WSRJaF/wUqjPNz+4prxwAAAABJRU5ErkJggg==","width":50,"height":50,"src":"/static/63581729799e31fcd1a0b0f4271c9cad/45876/avatar-big.png","srcSet":"/static/63581729799e31fcd1a0b0f4271c9cad/45876/avatar-big.png 1x,\n/static/63581729799e31fcd1a0b0f4271c9cad/eb85b/avatar-big.png 1.5x,\n/static/63581729799e31fcd1a0b0f4271c9cad/4f71c/avatar-big.png 2x,\n/static/63581729799e31fcd1a0b0f4271c9cad/9ec3e/avatar-big.png 3x"}}},"headerImage":null},"html":"Original AUTO1 architecture
\n\n
\n\n
\nResults
\n\n
\n
![](\"https://lh3.googleusercontent.com/FddBhrlKYHmv4Jw9mucH_Mzo0OUs7kbDEN2xOLqfPm7NCyNcM0fxD0MFSSQHODLDWYL8x-CrwCAj-nTJdeqbZGNk-5MrRwxrg7FI4ILuTQTQe8uEPG7OqHLrRZHnuhnjC9Os3x7KcBEDCaQWGJKk_amQo8GnfBh4PK87vfL6cgGD8buX61uiYEIclQ\")
![](\"https://lh6.googleusercontent.com/mEaBTCaWa1lWEc0cjQraEWVOPMb3V8StdxwrMvaALgxSxMBfjNjk4KBCRif-F_2oUVzgCBheGOpyLiAK5tnxPcH1m2Nae-8EMyi3CgOhoMUSSVAUJY8jFyckMLAq1PMhPI6UPE_uHSEeCy4yNrpX-mPx4he1VywC5lCtkwJBaC77Px36aqiMivkwig\")
![](\"https://lh4.googleusercontent.com/x_T21RH2LmaVQypUzhNgve4ZEPeKBTvWs0IXdJdPF_RVz3EXAb3SK09ZR1JRxmt6OUSTQdRpowIdPGUI0A17-ATiLE-nSY3u8hA0OjZZMO9HF-RCBw9q0I5GG_xIEwMyMy2g5OpxWc7QTmN2KnZy29jJ8pTdt0Av3MNQ4FuKS0e7FT3aRuYZv1diWA\")
![](\"https://lh3.googleusercontent.com/HsXSzSFwhynsa77tvxtyWg2K54pJpTXvXMghLmselpYCFLk7vktBkwWlQ5q7T_WpTiEYSTi85ULtxxz5fNhfleOVEhXf5NRm_51YPAh9x_t1aI53KyHs8QA1bRqLuhjMtZeSXwQIRrOq8rI2FxIxFs2bktIuLclo1ZgQN4sGEcdEKrrW-ZzE10tRHQ\")