{"data":{"allMarkdownRemark":{"edges":[{"node":{"id":"cfee7911-e765-5a6e-ba90-1eb2ce96d837","frontmatter":{"category":"Architecture","title":"Application Cockpit Model Outline","date":"2022-10-28","summary":"A short trip around the AUTO1 Application Cockpit data model","thumbnail":null,"authorName":"Przemysław Walat","authorDescription":"Przemysław is a Expert Software Engineer based in our Szczecin office","authorAvatar":{"relativePath":"pages/application-cockpit-model-outline/avatar_Walat.jpeg","childImageSharp":{"resolutions":{"base64":"data:image/jpeg;base64,/9j/2wBDABALDA4MChAODQ4SERATGCgaGBYWGDEjJR0oOjM9PDkzODdASFxOQERXRTc4UG1RV19iZ2hnPk1xeXBkeFxlZ2P/2wBDARESEhgVGC8aGi9jQjhCY2NjY2NjY2NjY2NjY2NjY2NjY2NjY2NjY2NjY2NjY2NjY2NjY2NjY2NjY2NjY2NjY2P/wgARCAAUABQDASIAAhEBAxEB/8QAGAABAQEBAQAAAAAAAAAAAAAAAAQDAQX/xAAWAQEBAQAAAAAAAAAAAAAAAAABAAP/2gAMAwEAAhADEAAAAYNcbctZHqE5AEDf/8QAGRAAAwEBAQAAAAAAAAAAAAAAAQIDEhEA/9oACAEBAAEFAnyRJdo0aBmiW9jE1khCKBOxPVY8/8QAGBEAAgMAAAAAAAAAAAAAAAAAABEBECH/2gAIAQMBAT8Beji//8QAFBEBAAAAAAAAAAAAAAAAAAAAIP/aAAgBAgEBPwEf/8QAHRAAAgIBBQAAAAAAAAAAAAAAAAERQQIgISIxgf/aAAgBAQAGPwJvpqi/CMJg4KTFN7l6P//EABsQAAMBAAMBAAAAAAAAAAAAAAABESExQWFR/9oACAEBAAE/IYU8XDGNg3v6Gar2x1WU9pKgnlpCG63tFjubo1NwXmn/2gAMAwEAAgADAAAAENfwvP/EABgRAAMBAQAAAAAAAAAAAAAAAAABEUFx/9oACAEDAQE/EHENHQ0h/8QAGREBAAIDAAAAAAAAAAAAAAAAAQAhEBFh/9oACAECAQE/EANUy+Z//8QAHRABAAIDAAMBAAAAAAAAAAAAAQARMUFRIWFxgf/aAAgBAQABPxB5rcKC843EOZKEWYB/dSvRVSN3XnMCCmUgo98hrV7b0d+xivco2sPKSmmz7hteDByVo6Gf/9k=","width":50,"height":50,"src":"/static/e4e77cd92050dbb448a088991f2d0c50/d2d31/avatar_Walat.jpeg","srcSet":"/static/e4e77cd92050dbb448a088991f2d0c50/d2d31/avatar_Walat.jpeg 1x,\n/static/e4e77cd92050dbb448a088991f2d0c50/0b804/avatar_Walat.jpeg 1.5x,\n/static/e4e77cd92050dbb448a088991f2d0c50/753c3/avatar_Walat.jpeg 2x,\n/static/e4e77cd92050dbb448a088991f2d0c50/31ca8/avatar_Walat.jpeg 3x"}}},"headerImage":null},"html":"<h1>I’m a model, <font color=“grey”><em>now</em></font> you know what I mean</h1>\n<p>    Welcome, fellow concerned developers. In our <a href=\"https://auto1.tech/application-cockpit/architecture-intro\">previous installment</a>, we have promised to delve deeper into the model that Application Cockpit is using underneath to see how it synergizes with the high-level architecture in order to make the whole product simple yet powerful and extensible. Be advised, that while we have always aimed for simplicity, our initial take on the model was a little bit more complicated, but after more-than-a-few heated debacles, we have stripped it down even more and so far we are pretty happy with it, despite more than a few \"I wish we had the initial design in place\" moments of weakness.</p>\n<p>    In honor of the early version, we are also going to take a short trip down the memory lane to see the trade-offs and compromises we have decided to make along the way and features we have decided to add. We are going to discuss the direction we took and what were our reasons to do so (or in other words, \"what were they thinking ?\"). But for now...</p>\n<h2>Where we're at</h2>\n<p>    As we hinted in the previous blog post, Application Cockpit is actually binary, in a sense that its primary domain consists of only two data types: Resources and Readings and the (extremely) short story is that a Resource is an entity for which we are going to track Readings, while a Reading is a piece of information pertaining to a given Resource. The longer story follows below.</p>\n<h3>Resources</h3>\n<p>    Like mentioned above, a Resource is mostly something that we are going to collect readings for, it can be a physical entity, or a purely logical one. Examples of currently tracked Resources include:</p>\n<ul>\n<li>\n<p>(semi)physical</p>\n<ul>\n<li>\n<p>LambdaDeployment</p>\n</li>\n<li>\n<p>RdsInstance</p>\n</li>\n<li>\n<p>ServiceDeployment</p>\n</li>\n<li>\n<p>ServiceInstance</p>\n</li>\n</ul>\n</li>\n<li>\n<p>logical</p>\n<ul>\n<li>\n<p>Lambda</p>\n</li>\n<li>\n<p>Service</p>\n</li>\n<li>\n<p>SoftwareDependency</p>\n</li>\n<li>\n<p>SoftwareDependencyVersion</p>\n</li>\n</ul>\n</li>\n</ul>\n<p>The Resource has a very rigid model, comprising of only a few, well-defined properties, that is:</p>\n<ul>\n<li>\n<p>id -  the unique UUID</p>\n</li>\n<li>\n<p>type - the type of the resource, see above for examples</p>\n</li>\n<li>\n<p>name - a textual identifier, unique across one type; needs to be deterministically derived from resource's properties available at runtime (further details in tradeoffs chapter)</p>\n</li>\n<li>\n<p>parent_id - id of the parent resource (if any)</p>\n</li>\n<li>\n<p>created_on - a timestamp and a late-comer to the party, but we find it useful in many ways, especially during troubleshooting</p>\n</li>\n</ul>\n<p>In order to publish (a.k.a create) a Resource, the collector will provide only the name, type and optionally the parent identifier.</p>\n<p>When it comes to type, it is a free-form String. We have no predefined enumeration to represent it and the collectors (see previous article to learn more about them) are free to push whatever types they want, akin to the front-end which is allowed to ask for any type it wishes.</p>\n<p>In addition, the inclusion of parent id allows us to build hierarchies, so that our data closely follows the domain that we are modeling.</p>\n<br>\n<h3>Readings</h3>\n<p>    Given the above, you should now have a general idea of what a Resource is and how we use it, so we can move on to Readings. Since we have established that it is generally a piece of data pertaining to a Resource, let's have a look at its properties:</p>\n<ul>\n<li>\n<p>id - the serial identifier;</p>\n</li>\n<li>\n<p>type - type of the Reading; free form String, akin to Resource type</p>\n</li>\n<li>\n<p>resource_id - uuid of the Resource this Reading describes</p>\n</li>\n<li>\n<p>value - free-form json document to hold the data of the Reading</p>\n</li>\n<li>\n<p>storage<em>mode - determines the behavior when multiple Readings are published with same resource</em>id and type:</p>\n<ul>\n<li>\n<p>SNAPSHOT - only the most recent Reading is preserved</p>\n</li>\n<li>\n<p>TIME_SERIES - all of the published Readings are stored</p>\n</li>\n</ul>\n</li>\n</ul>\n<p>Since the values we store for Reading are free-form JSON, it gives developers a lot of flexibility in how to handle their flows. We store both trivial things, e.g.:</p>\n<p><img src=\"https://lh5.googleusercontent.com/v1PbZaRZQJjKJbBQiQZOXRUGjLK_c000j3edWpNb7jhim2dDHl21ircQVV4-aabJnHA57-A68J7WXfTkMFFxeVro0kGAlSMh2PKZ72yW5IezNvN-FiM4Xm-siC7PhAS8ZarnG4imewzKJhjKxAR7Mt0A3R5BOkpSCDYW14ldqG2CLpW5XC-jbwJtvg\"></p>\n<p>To the humongous too-big-to-fit-in-the-picture guys:</p>\n<p><img src=\"https://lh3.googleusercontent.com/FddBhrlKYHmv4Jw9mucH_Mzo0OUs7kbDEN2xOLqfPm7NCyNcM0fxD0MFSSQHODLDWYL8x-CrwCAj-nTJdeqbZGNk-5MrRwxrg7FI4ILuTQTQe8uEPG7OqHLrRZHnuhnjC9Os3x7KcBEDCaQWGJKk_amQo8GnfBh4PK87vfL6cgGD8buX61uiYEIclQ\"></p>\n<p>Again, akin to Resource type, the Reading's type and value are completely opaque to most of the Application Cockpit infrastructure, so individual contributors are free to ship anything they wish in those properties. The only components which need to be aware of the exact structure are the Collector that is originally publishing the data and the UI component which is querying the Data Provider in order to render the respective view. That way, many developers are free to add their own custom extensions, without stepping on each other's toes or needing to seek approval or assistance from the core Application Cockpit team, which in addition can remain blissfully unaware of the atrocities that the others may commit and attempt to hide within their custom Readings.</p>\n<br>\n<h2>The tradeoffs</h2>\n<p>    Having dealt with the detailed model description, we can now move on to discuss how our journey with its design looked like. Of course we, being the pragmatic, more-than-once-bitten little members of the software engineering club that we are, were under no delusions about what is going to happen to our nifty design when it gets into its first bar fight against the necessities of real life. The only question was when we are going to be faced with our first moral/design dilemma and how much are we going to be willing to compromise our collective conscience.</p>\n<br>\n<h3>The Resource id conundrum</h3>\n<p>    When it comes to the little trade-offs, actually, the more observant reader might have noticed by now that something is a bit amiss in our picture:</p>\n<ol>\n<li>\n<p>In this article we have learned that in order to publish a Reading for a Resource, the Collector probably needs to provide the resource_id to bind it to the right entity</p>\n</li>\n<li>\n<p>In the previous article we have mentioned, that Collectors are generally stateless and have no access to the storage holding Resource data</p>\n</li>\n<li>\n<p>In consequence, how can a Collector find out the correct id to publish the reading for ?</p>\n</li>\n</ol>\n<p>Well, it  would that it could, but it can’t, so it shan’t. It’s quite a bind we find ourselves in, right ? We were adamant about not allowing collectors access to our DB, so we had to find another way to make things happen… A short look at the ReadingPublishMessage model, might provide some hints, though, so let’s start with that:</p>\n<p><img src=\"https://lh6.googleusercontent.com/mEaBTCaWa1lWEc0cjQraEWVOPMb3V8StdxwrMvaALgxSxMBfjNjk4KBCRif-F_2oUVzgCBheGOpyLiAK5tnxPcH1m2Nae-8EMyi3CgOhoMUSSVAUJY8jFyckMLAq1PMhPI6UPE_uHSEeCy4yNrpX-mPx4he1VywC5lCtkwJBaC77Px36aqiMivkwig\"></p>\n<p>Does anything stand out to you ? It is almost a reflection of a Reading, but with a slight twist, and you probably got it right, so enter  Resource's name - that tiny little property looks like it has been put there in order to provide a human-friendly name for a Resource, but it couldn't be further from the truth than that. In fact, initially it was never there (we use Readings to store human-friendly names of the Resources anyway) and we added it specifically to solve the issue at hand.</p>\n<p>If you recall, the description says that it needs to be unique and possible to deterministically assemble from the data available in the runtime. Leveraging that, the Collector doesn't need to know the right id, it just needs to provide the correct textual name. The good news is that we have a way of overcoming the initial obstacle, but there is also bad news... This forces every collector to be aware of the logic required to construct names of the Resources it is interested in, moreover, the logic needs to be in multiple places and it needs to be synchronized. But let's leave it at that, since data ingestion is going to be the focal point of the next article in series.</p>\n<br>\n<h3>Can I haz a Resource ?</h3>\n<p>    At this point a meticulous reader with a penchant for domain modeling might begin to wonder:</p>\n<p>\"All right, a Resource can have a parent... That's nice, but is that really all that there is? You say you want to closely match the domain, but not everything is a child of something, sometimes you might want a Resource to 'own' Resources, so... how about that, smarty pants ?\"</p>\n<p>This was also something we were intensively brainstorming about and lots and lots of coffee cups later (and probably more than a few beer mugs, too) we have arrived at something we were not very uncomfortable with. Our main priority was to avoid complicating the model (remember ? we want people to integrate easily, so we want to keep things simple), so we didn't want to introduce any new concepts or tables. Instead, we elected to cheat a little, so please have a look at the example below with a picture being worth a thousand words:</p>\n<h2><img src=\"https://lh3.googleusercontent.com/9ofjHjm_CHNEKWH-7C6kdE5aq9ELbeH4ekpOoOZcDMJtwYjP3xWOTiceE3kW1i0pl4H3uTCbh0c5eY1MMqpWJRXjB0J4L66x0vf6YHhs15fY-s_oA-8qobF_eynejn8xlsns09OlmgIGAy7bZNDDebFodPEHmGrPgp8HnAhoCCUIzEi7fV1OWheP\"></h2>\n<p>Basically we have elected to model a \"has a\" relationship between two Resources using an intermediary Reading to store ids of the referenced (one or many) Resources. In our example a Service Deployment has a Reading Called \"Service Deployment Dependencies\". The main purpose of this Reading is to hold data which we can use to construct id of \"Software Dependency Version\" Resources that our \"Service Deployment\" has. As you can see, if we wanted to obtain vulnerabilities of all dependencies a specific Service Deployment Resource has, we could easily construct ids of each and every \"Software Dependency Version\" and then query for its \"Dependency Version Vulnerability\" Readings. </p>\n<p>There aren't any places where we use this not-so-orthodox approach, but there are some and the example below is the most prominent real use-case. We are closely monitoring the performance impact it has on our service and so far, it has been negligible, or at least negligible enough that we are still able to look at ourselves in the mirror.</p>\n<br>\n<h2>and the (mostly) harmless evolution</h2>\n<p>    Having confessed our most prominent sins, let's wrap things up with a less controversial story. There are several items that were not part of the initial design, which we have decided to add along the way, because we have found them useful.</p>\n<br>\n<h3>To replace or... not to replace</h3>\n<p>    The storage mode for a Reading was a last-minute addition to the picture, but it serves its purpose well. Initially we thought of Readings solely as means of representing the current state of things, e.g. the single-point-in-time data like:</p>\n<ul>\n<li>\n<p>tech stack given service is using</p>\n</li>\n<li>\n<p>AWS account id of the deployment</p>\n</li>\n<li>\n<p>IP address of an instance, etc.</p>\n</li>\n</ul>\n<p>This meant that we were only interested in the most recent value of a specific Reading Type for a Resource, always replacing it whenever a new value was collected. From the more technical standpoint - we could get away with performing upserts on Readings with the key being comprised of resource<em>id and reading</em>type. However, after a few initial releases, we've discovered that there are certain pieces of data more linear in their nature, which Application Cockpit users could really appreciate on a daily basis.</p>\n<p>A prominent example of such an item is Service Events Graph, which as you can probably guess represents a service's life cycle timeline, so a mixed bag of Service Deployments, Instance Startups and Shutdowns, Incidents etc. Here's what it currently looks like:</p>\n<p><img src=\"https://lh4.googleusercontent.com/x_T21RH2LmaVQypUzhNgve4ZEPeKBTvWs0IXdJdPF_RVz3EXAb3SK09ZR1JRxmt6OUSTQdRpowIdPGUI0A17-ATiLE-nSY3u8hA0OjZZMO9HF-RCBw9q0I5GG_xIEwMyMy2g5OpxWc7QTmN2KnZy29jJ8pTdt0Av3MNQ4FuKS0e7FT3aRuYZv1diWA\"></p>\n<p>What we have also found out rather quickly was that it would be quite awkward to represent this kind of data using our initial point-in-time-based approach to Readings.</p>\n<p>Enter storage mode - a property of a Reading which tells the underlying infrastructure how to behave when a new Reading for the same resource<em>id and reading</em>type is persisted. </p>\n<p>To preserve all the already existing flows, we have assumed the default mode to be SNAPSHOT, which upholds the default behavior of upserting new values, keeping all the implemented features intact without any additional effort. </p>\n<p>As an extension of the functionality, we have introduced TIMESERIES mode, which preserves all the values that were collected and allows us to construct e.g. neat timelines like what you can see in the screenshot above. </p>\n<p>As a bonus, here's how the actual data underlying the graph looks like:</p>\n<p><img src=\"https://lh3.googleusercontent.com/HsXSzSFwhynsa77tvxtyWg2K54pJpTXvXMghLmselpYCFLk7vktBkwWlQ5q7T_WpTiEYSTi85ULtxxz5fNhfleOVEhXf5NRm_51YPAh9x_t1aI53KyHs8QA1bRqLuhjMtZeSXwQIRrOq8rI2FxIxFs2bktIuLclo1ZgQN4sGEcdEKrrW-ZzE10tRHQ\"></p>\n<h3>A little housekeeping never killed nobody... I hope</h3>\n<p>    Whilst it seems quite natural for a Reading to have a timestamp since it is collected (or, as you will, measured) at a specific point in time, it's not so straightforward in case of a Resource, especially not from the Business Domain side of things, since it mainly represents a certain Entity that simply is.  Seems like we're not all about Business, I guess, but we'll get to that in a while.</p>\n<p>With the above in mind, the first iterations of the Resource model did not have a good reason to include timestamps, so they didn't. However, as we kept adding new features and collectors, we have started noticing that the system is accreting quite a lot of Resources and that a rather large portion of them is only relevant within a short timeframe. It wasn't a problem for us at that point, but even though we hate doing the house chores as much as we do, why not keep our room tidy and \"let go of the things that don't matter anymore\", as we've all certainly been told at least a few times in our life ?</p>\n<p>The part where it most stood out for us were Service Instance resources which by their nature are short lived, since as soon as the instance is dead, we don't care about it anymore (apart from audit reasons perhaps). The point was - how do we know whether we can safely delete the Instance Resource ? </p>\n<p>As you have seen above, we have ServiceEvents telling us when instances were terminated, but can we rely on the event to always be generated and are we willing to trust it ? Not really.</p>\n<p>Alternatively, we do track instance health checks but they are not ideal by themselves, particularly since a freshly minted Instance would have no checks at all in the beginning and then report as unhealthy for some time before it becomes stable, not to mention that in some cases it is all right for an instance to fail some of the inspections for a while and still survive. The list of things to consider goes on and on, which went contrary to our goal which was to avoid creating a too-sophisticated cleanup-logic for fear it would become too brittle to be effective.</p>\n<p>The introduction of Resource timestamps allowed us to reach all the objectives in a relatively low-cost and simple manner. Once we started tracking when an Instance was created it was a simple manner scanning Instance Resources on a schedule, discovering those which had no health reports for quite a while and were not created recently to safely trim them away from our storage.</p>\n<p>The added bonus is that having a Resource creation timestamp comes in handy during investigation of issues (not that there are too many), since it makes it easier to track the timeline and wrap your head around what happened when and why.</p>\n<br>\n<h2>That's all, folks</h2>\n<p>    At this point I am going to conclude my musings and I hope that you've found them interesting or perhaps even inspiring at some point. The topic is far from exhausted, especially given that our journey with Application Cockpit is still well under way and we keep on adding new features into the application, so please look forward to the next article in series.</p>","fields":{"slug":"/application-cockpit-model-outline/","tags":["auto1","Cockpit","DataModel","Architecture","Core"]}}},{"node":{"id":"4c23b9b2-0718-52ff-a466-8b87fec6925d","frontmatter":{"category":"Architecture","title":"Machine Learning pipeline","date":"2022-08-31","summary":"Machine learning systems are complicated, but good infrastructure can reduce this complexity by automating processes. Well thought out infrastructure solutions can lower the engineering time and accelerate your teams ability to deliver ML applications. In this blog post, I go over an example of automating a ML pipeline using AWS services, review the structure and give you some action points to consider when creating your own.","thumbnail":null,"authorName":"Alena Nazarava","authorDescription":"Alena Nazarava is a Data Scientist working remotely from Bavaria","authorAvatar":{"relativePath":"pages/machine-learning-pipeline/avatar_Nazarava.png","childImageSharp":{"resolutions":{"base64":"data:image/png;base64,iVBORw0KGgoAAAANSUhEUgAAABQAAAAUCAIAAAAC64paAAAACXBIWXMAAAsSAAALEgHS3X78AAAEgUlEQVQ4yyXRaWwUZRgH8EnEkxTXcrVFzpJSIUBtqdB272Nm53xn3pnZOXZmdmZn9u6W0m5blkKtUIkCtQiklOOL1MolEEBDBQP1SEwUNcaEeEXRT4qJH4x+aiJuNXnyfPs9/+T/ILIiSBKURSDxpASJGMRFFhfYqERjMkAVJsJhHoHwiyDMM0ERhHhQ2WEFYCrAkTks8zGRE3lK4HABRHkGjzFRlSF41P9cjatu/rz6RQtI/zaVQ2U6HGMiMojIEFO4/7EEeQFwHAUZjKMikEJFMqzQGLqpvhxp1FqWzUeQtbULFRBR6YgydzeqcngckoiiirEYFDiGAyRHoSwRAtEARANoW/OIFng4c2D27MB+6Kt5cp4Q9WpsRKZCCo3GWVyH1H9Y4gTIQICzJArQIAi6uZDH29R4plf+/mjXr+OF6e1861IXCLXpHKbQIQ2gOoslBApR4xUMBZ6GLMESERDxg0AHG+7wNa17e8j5c3L3g/HinZLqrnMxnhaDxys+AaOmSFoSU8ExReZlAYgczVEYQH1MyMMF2wMtGw4n8L+m9j4YzV028c1Vj9DuzRVswGgyRtky4ygA0XRFU2OqxEkQCDQBsSAI+dmQh2xvZpvX/jJ14IcRm1heve4JhGxdbwpEZWyJcWQmpbGIXsG6rKliXGQljhYoDEb8pN9NtG9pXVH75cnShURHU9W8zQse9TSssGU6GaMdicmoXNaASCKhGUbc0BRdEeIiUDiSRf39cjQDqRFHnL09OtOLMfWuTdWPN7iqFCacUumUwmR1Lm8KSNI2kqZm6IqhyYYCFUBIhP/GWM/0+clPbt+cPvnauZGSZ9XCjYuekhuWHoz7KuF5XchbQtGWENu2kpZhJuIVr8ucwuKVtq9NvHTvq7vnLr5z5eLV0f7iOtdjgZXPnNHd34zqOYkqJKWiI3WnVcRxbNs2rblwWeVpmSFxd9urw8Off3r3i49nThw9zW5t6lhTJzYuvJbx3j+eLPDh7Sm1Jx3vzWpIOuXYjmVaWqVzGQI+irGhwJtvTL5/68OP7sz0pbM6gUc3rmFWPf1u1nP/dbUoBLszWiln9BUsJJV2ko5pWYmErqoCB6MYj4Y/++DWW5fem5w4PrGrc0cm317r8tdUndWe/208WTZBVyY+0JkodztIJp/J5jLptGMauirwkMRJb8fLI4cuX705NT42NXGEcnt9y6o9tVU3iv7ZSwP78rEdebPcZe8pZZF8MV/ozKVStlGpW2AhTUTat3V3dv/843dnL1w9eOgYumG9/9nFVOPy+8esf64MHOoxejqtod7s0M4ikitk09mUnbR0XZV4AAgMD3hZX3j/zlcOj57aUxp0r65vq1sS9239/Xzfw+uDY/1WqWgP93cODXYhc7HZVNKxdEMRIQA4GnRvdWQ9K+TQTQH36uYtNWs2Vi+Wg94/rgzNXt91sJQYKKZfHOgeHuxF0l2FVCZlOmbcUCHPUATqfqF1b3n3va+/PX3qwokjlVf5W5asDKxd/dNk6e/p4X09dnlHYVdf155y6V+to7XKa9wQBAAAAABJRU5ErkJggg==","width":50,"height":50,"src":"/static/9ff0aff34e80697669a1119f76326bbe/45876/avatar_Nazarava.png","srcSet":"/static/9ff0aff34e80697669a1119f76326bbe/45876/avatar_Nazarava.png 1x,\n/static/9ff0aff34e80697669a1119f76326bbe/eb85b/avatar_Nazarava.png 1.5x,\n/static/9ff0aff34e80697669a1119f76326bbe/4f71c/avatar_Nazarava.png 2x,\n/static/9ff0aff34e80697669a1119f76326bbe/9ec3e/avatar_Nazarava.png 3x"}}},"headerImage":null},"html":"<h1>Machine Learning pipeline</h1>\n<h2>Introduction</h2>\n<p>Developing a machine learning pipeline generally requires a custom approach. There is no off the shelf, well-architectured system that will work for all use cases. On the contrary each use case requires you to design the structure that will satisfy your particular requirements. The process for developing, deploying, and continuously improving them is quite complex. There are three main axes that are subject to change: the code itself, the model, and the data. Their behavior can be hard to predict, hard to test, explain and improve.</p>\n<p>There are some aspects that are important to know before you start building the system. For example:</p>\n<ol>\n<li>Will you be doing online or batch predictions?</li>\n<li>How often will you need to train the model?</li>\n<li>Will your pipeline need to be fully or partially automated?</li>\n</ol>\n  <p>&nbsp;</p>\n<h2>Use Case</h2>\n<p>For this example, we will take the use case of making vehicle recommendations for our Newsletter. The newsletter is sent out once per day and will contain a set of recommended vehicles. As additional information, we have a lot of new items added to our inventory each day. Therefore our system will have better performance if we train the model more often. Since the newsletter is daily, we can make a single prediction and have the data ready for the newsletter to use.</p>\n<p>And of course, rather than perform these steps each day, it makes sense to automate this process and save our time for more exciting challenges. With this in mind, we can start answering some questions and determine which components we need and what the general structure of the pipeline will be.</p>\n  <p>&nbsp;</p>\n<h2>Overview</h2>\n<p>Even with a described use case and defined requirements, there can be many possibilities for design structures. We are extensively using AWS as a cloud provider. So, we will use SageMaker to automate training our model, and as storage for the temporary results, an S3 Bucket. Data preprocessing is a separate step and will be implemented outside of SageMaker. The final consideration is how we will deliver the predictions to other teams.</p>\n<p><a\n    class=\"gatsby-resp-image-link\"\n    href=\"/static/091f446f1b671b13b7f953faa9eb2f90/9da67/IMG_1.png\"\n    style=\"display: block\"\n    target=\"_blank\"\n    rel=\"noopener\"\n  >\n    <span\n    class=\"gatsby-resp-image-wrapper\"\n    style=\"position: relative; display: block;  max-width: 590px; margin-left: auto; margin-right: auto;\"\n  >\n    <span\n      class=\"gatsby-resp-image-background-image\"\n      style=\"padding-bottom: 41.705069124423964%; position: relative; bottom: 0; left: 0; background-image: url('data:image/png;base64,iVBORw0KGgoAAAANSUhEUgAAABQAAAAICAYAAAD5nd/tAAAACXBIWXMAAAsSAAALEgHS3X78AAABlElEQVQoz3WSXW7iMBSF2QfbYBWUPUw1bKKdVfEE4g1eIEhMVQZNoCGFGELCJCH/P05aesbXEa2qqg9Hjj/7nnt8lQY7hnjaOmBHH4bpY3c4f9F3nLRlnlg9rHQHe+HRMO0QSZKgyFPEUShXznPwIkPJC/mdJnHNJL8qQ1Vx4O0C4AI/iLBj/4ThKYJ3DrBYLKAoM2jaBnkuTFLRII4xmUwxGAywXC5RFFyygnMYBsOP21vc3HQwnU4RJQW2xgmNozB03DMeHn5DVVWs12tkWSbScFiWJc36/T7G4zFc15VnVVXJBp1OB81mE93uT/DygmfDrg0p4Ww2k500TZNFlNL3fQyHQ/R6PYxGI8lpPGVZQtd1tNtttFot/Lq/Q5pXwvBUzzCMEpimKZ9NKcgsSVOZkjGG+XwO27blnkZBojuKosgXPOsbnP0Yu72Y4d4KEIQxoiiSppSAklARrZ7nSh4EwTu/nlEDqnt9fYHjhXVC+l3+qAxPugVta2O1MbHSPnTl683xEyep2kHcr/njXwZ2cPAfYfNAaR92sB8AAAAASUVORK5CYII='); background-size: cover; display: block;\"\n    ></span>\n    <img\n        class=\"gatsby-resp-image-image\"\n        style=\"width:100%;height:100%;margin:0;vertical-align:middle;position:absolute;top:0;left:0;box-shadow:inset 0px 0px 0px 400px white;\"\n        alt=\"IMG 1\"\n        title=\"\"\n        src=\"/static/091f446f1b671b13b7f953faa9eb2f90/40fad/IMG_1.png\"\n        srcset=\"/static/091f446f1b671b13b7f953faa9eb2f90/707e9/IMG_1.png 148w,\n/static/091f446f1b671b13b7f953faa9eb2f90/649e0/IMG_1.png 295w,\n/static/091f446f1b671b13b7f953faa9eb2f90/40fad/IMG_1.png 590w,\n/static/091f446f1b671b13b7f953faa9eb2f90/9da67/IMG_1.png 868w\"\n        sizes=\"(max-width: 590px) 100vw, 590px\"\n      />\n  </span>\n  </a></p>\n <figcaption align = \"center\"><sub><b><em>Main components</b></figcaption>\n<figcaption align = \"center\"><sub>source: draw.io</em></figcaption>\n<br>\n<h2>Pipeline Components</h2>\n<p>Let's describe each component in more detail, starting with the model. Ideally we would want the model to continue training on new data, rather than from scratch each time. This continual learning will allow us to update the model with less data. Instead of loading data from the database for the last few months, it will only need to fetch data from the previous day. This incremental approach of training the model will need much less compute power. It will also help us to avoid storing duplicate data, as each model update is trained using only fresh data.</p>\n<p><a\n    class=\"gatsby-resp-image-link\"\n    href=\"/static/e0b32c3e2f366966fab1c9826ca5beb8/86db2/IMG_2.png\"\n    style=\"display: block\"\n    target=\"_blank\"\n    rel=\"noopener\"\n  >\n    <span\n    class=\"gatsby-resp-image-wrapper\"\n    style=\"position: relative; display: block;  max-width: 590px; margin-left: auto; margin-right: auto;\"\n  >\n    <span\n      class=\"gatsby-resp-image-background-image\"\n      style=\"padding-bottom: 38.61967694566813%; position: relative; bottom: 0; left: 0; background-image: url('data:image/png;base64,iVBORw0KGgoAAAANSUhEUgAAABQAAAAICAYAAAD5nd/tAAAACXBIWXMAAAsSAAALEgHS3X78AAABl0lEQVQoz01Sy3LaQBDk//8k5xySSlJ24oCrTJUJYIEFSBFgad+rB7K0yD50ZjdQyaFrZnpne2daGuXM4JBrMFlhf5RgosJLYXAkjssahxeFnFsUogznnvNnvoerhlBD6gbHwkKZFqM02WI6fYC1BpPJGKzIsdtt8Wv2CGs0Fos5NpsYUgqMxz9hiIuip8C7voNzPc6EqmpIuCbBNCHBKRElCU7AGUOaJCQ4I67CjGIce0EZBLXWWK0iLOZzNM0JbfuK89mhDIINRt/v1/h0s8TXu4iwwpcfEeEJn2+X+DZe42YS4/Z+E3DNfbx72OLDx0c87wpoJXA45tCWVhZSgQvyqeBI0ixEXzP+lzO2gpDkJ9XG1sRLSGVo0jJEP+Fr20Ibqg1NONC4zjmcTg2E4NTQXnxxtFKDLPsNpWRY2efeBs4Z9lkW7nRdh77vYcvLysPgiOiCD2/DEAx21HAV9Y/0dKltT0Ho/f0NdV2FjzQM59Drh6rJT+UnjHc5rnj+L79ik7JLXoQ89CSUE/71FFiu9/R7lfgDorNKPr2LIv4AAAAASUVORK5CYII='); background-size: cover; display: block;\"\n    ></span>\n    <img\n        class=\"gatsby-resp-image-image\"\n        style=\"width:100%;height:100%;margin:0;vertical-align:middle;position:absolute;top:0;left:0;box-shadow:inset 0px 0px 0px 400px white;\"\n        alt=\"IMG 2\"\n        title=\"\"\n        src=\"/static/e0b32c3e2f366966fab1c9826ca5beb8/40fad/IMG_2.png\"\n        srcset=\"/static/e0b32c3e2f366966fab1c9826ca5beb8/707e9/IMG_2.png 148w,\n/static/e0b32c3e2f366966fab1c9826ca5beb8/649e0/IMG_2.png 295w,\n/static/e0b32c3e2f366966fab1c9826ca5beb8/40fad/IMG_2.png 590w,\n/static/e0b32c3e2f366966fab1c9826ca5beb8/b3fef/IMG_2.png 885w,\n/static/e0b32c3e2f366966fab1c9826ca5beb8/301c0/IMG_2.png 1180w,\n/static/e0b32c3e2f366966fab1c9826ca5beb8/86db2/IMG_2.png 1362w\"\n        sizes=\"(max-width: 590px) 100vw, 590px\"\n      />\n  </span>\n  </a></p>\n<figcaption align = \"center\"><b><em><sub>Incremental training</b></figcaption>\n<figcaption align = \"center\"><sub>source: draw.io</em></figcaption>\n<br>\n<p>We are using SageMaker for training. We constructed it by loading a SageMaker Pytorch image, optimized for GPU and customized with our training configuration. During deployment of a new version, the image is pushed to ECR so that SageMaker can fetch the proper image and then spin it up inside our SageMaker space. After training SageMaker saves the resulting artifacts to an S3 bucket. But Sagemaker won’t include the ETL part and we need to put in place the data loading and processing part separately. For this purpose we will have a dedicated job in Glue. It’s a serverless data integration service for performing ETL workloads without managing any underlying servers.</p>\n<p><a\n    class=\"gatsby-resp-image-link\"\n    href=\"/static/ea172fd544b3e1400c1ee9ab71e6a3f3/1ce75/IMG_3.png\"\n    style=\"display: block\"\n    target=\"_blank\"\n    rel=\"noopener\"\n  >\n    <span\n    class=\"gatsby-resp-image-wrapper\"\n    style=\"position: relative; display: block;  max-width: 590px; margin-left: auto; margin-right: auto;\"\n  >\n    <span\n      class=\"gatsby-resp-image-background-image\"\n      style=\"padding-bottom: 25%; position: relative; bottom: 0; left: 0; background-image: url('data:image/png;base64,iVBORw0KGgoAAAANSUhEUgAAABQAAAAFCAYAAABFA8wzAAAACXBIWXMAAAsSAAALEgHS3X78AAABSUlEQVQY0zWQy0rDUBCG+xI+hwsfwYX6DL6BCxHcihtFQRDcKLgQpCIoIgiKItVqFRGxVitqL1DbNElzcnJi0zbUxi4+j6EuPoZ/Lj8zkzDsL6RqI2QQR1e1cKXGG6B1nB9o4QUx//q/JnVfqSJIKL+DCgSZlxQNVcUshByt+qR3Jf1en/TbEUunU9wXU0TdiGT+joXbE0qOTbMpWL+aZ+tmmaCtqBryzzDElDX20zsU63mergSzY1lWp5/ohRFrJ4sMzw2RvNykE3SZOdxmdGOR2493qvUyEysjTK6P43gNbeiRiNeVDvlCDse1UW6L9PErLw+f9KOIXOGRw+s9XkvP/PQiMvksB5mUNpD0oi4Xj2dcZs/5/g6p1PTJhqWo1V3MQbQcD0cpHOljmBLL9pFuGNcNSyJEE6mxbIWpcYX+odvWcz7lSoNfiJ9eulb9R3MAAAAASUVORK5CYII='); background-size: cover; display: block;\"\n    ></span>\n    <img\n        class=\"gatsby-resp-image-image\"\n        style=\"width:100%;height:100%;margin:0;vertical-align:middle;position:absolute;top:0;left:0;box-shadow:inset 0px 0px 0px 400px white;\"\n        alt=\"IMG 3\"\n        title=\"\"\n        src=\"/static/ea172fd544b3e1400c1ee9ab71e6a3f3/40fad/IMG_3.png\"\n        srcset=\"/static/ea172fd544b3e1400c1ee9ab71e6a3f3/707e9/IMG_3.png 148w,\n/static/ea172fd544b3e1400c1ee9ab71e6a3f3/649e0/IMG_3.png 295w,\n/static/ea172fd544b3e1400c1ee9ab71e6a3f3/40fad/IMG_3.png 590w,\n/static/ea172fd544b3e1400c1ee9ab71e6a3f3/b3fef/IMG_3.png 885w,\n/static/ea172fd544b3e1400c1ee9ab71e6a3f3/301c0/IMG_3.png 1180w,\n/static/ea172fd544b3e1400c1ee9ab71e6a3f3/1ce75/IMG_3.png 1456w\"\n        sizes=\"(max-width: 590px) 100vw, 590px\"\n      />\n  </span>\n  </a></p>\n<figcaption align = \"center\"><b><em><sub>Pipeline</em></b></figcaption>\n<figcaption align = \"center\"><em><sub>source: draw.io https://aws.amazon.com/architecture/icons/</em></figcaption>\n<br>\n<p>Individual AWS services are powerful tools by themselves, but chaining their outputs to the inputs of others services can be challenging without step functions. Step Functions coordinate different pieces of the pipeline and control which steps execute and when. It has a json format that represents a structure. States is an object containing a set of state/steps. The Step Functions service (SF) coordinates the order of execution.</p>\n<p><a\n    class=\"gatsby-resp-image-link\"\n    href=\"/static/f23cebe966e6a2c802df6140a532ee19/6d785/IMG_4.png\"\n    style=\"display: block\"\n    target=\"_blank\"\n    rel=\"noopener\"\n  >\n    <span\n    class=\"gatsby-resp-image-wrapper\"\n    style=\"position: relative; display: block;  max-width: 590px; margin-left: auto; margin-right: auto;\"\n  >\n    <span\n      class=\"gatsby-resp-image-background-image\"\n      style=\"padding-bottom: 59.78260869565217%; position: relative; bottom: 0; left: 0; background-image: url('data:image/png;base64,iVBORw0KGgoAAAANSUhEUgAAABQAAAAMCAYAAABiDJ37AAAACXBIWXMAABYlAAAWJQFJUiTwAAABZUlEQVQoz5VT2W6DMBDk/z8wT80hIKQBDMY24AMy3XUCIkqrpJZWrC00nmOdWGvhvQd/Q/BwXmO0HabJg9ftdov16UqmacLpdIqltYYZaqj+Ch/sCvauthcnvGGgLMvQthLWDcQu4D9rqyBZmv3+C3l+Ri3OqJoDdF8R2yqyHWwbv7ovocw3WpVSf419CO6VITdlWSJNU1SVQD8oDGNLXkr0Y0O+GrJgRJgcAZDnYaDexrN5np8BF7ose7fb4Xg4QnYCjcwfrK5kg/5c8rLhm5RSuFwKFEUBYwyxE5AqjxJblRFr+cTm11C2gM451DWlrHSUxIl3pkCnC5LdbwD+HqXVw67j2WM/ZgJ3D3+mF2nvKgLyUAtRE0NPicqYZmcuccD/uyIg+1VVVSwhGgrBxpcyzT4myj0ztpQ2WxGo53M+YyuWnhWtkhdATptZMsP73N2Z8swJeaSREqsCngKp8/jP8rp+ANvEqYNckCZwAAAAAElFTkSuQmCC'); background-size: cover; display: block;\"\n    ></span>\n    <img\n        class=\"gatsby-resp-image-image\"\n        style=\"width:100%;height:100%;margin:0;vertical-align:middle;position:absolute;top:0;left:0;box-shadow:inset 0px 0px 0px 400px white;\"\n        alt=\"IMG 4\"\n        title=\"\"\n        src=\"/static/f23cebe966e6a2c802df6140a532ee19/40fad/IMG_4.png\"\n        srcset=\"/static/f23cebe966e6a2c802df6140a532ee19/707e9/IMG_4.png 148w,\n/static/f23cebe966e6a2c802df6140a532ee19/649e0/IMG_4.png 295w,\n/static/f23cebe966e6a2c802df6140a532ee19/40fad/IMG_4.png 590w,\n/static/f23cebe966e6a2c802df6140a532ee19/b3fef/IMG_4.png 885w,\n/static/f23cebe966e6a2c802df6140a532ee19/301c0/IMG_4.png 1180w,\n/static/f23cebe966e6a2c802df6140a532ee19/6d785/IMG_4.png 1288w\"\n        sizes=\"(max-width: 590px) 100vw, 590px\"\n      />\n  </span>\n  </a></p>\n<figcaption align = \"center\"><sub><b><em>Example state machine code</em></b></figcaption>\n<br>\n<p>By default SF sends the output of a previous state as the input of the following state. It provides service integration, which means it can connect and use other AWS services as steps. During execution there is an available context object, which contains info about the state machine and execution. This allows the workflows access to information about a specific execution. There is also an execution log, where you can find info about tasks and time spent in each step. This can be helpful for debugging or tuning your execution.</p>\n<p><a\n    class=\"gatsby-resp-image-link\"\n    href=\"/static/002e2cabf19bf7c3dd2b5c1d138f1f43/7acd2/IMG_5.png\"\n    style=\"display: block\"\n    target=\"_blank\"\n    rel=\"noopener\"\n  >\n    <span\n    class=\"gatsby-resp-image-wrapper\"\n    style=\"position: relative; display: block;  max-width: 590px; margin-left: auto; margin-right: auto;\"\n  >\n    <span\n      class=\"gatsby-resp-image-background-image\"\n      style=\"padding-bottom: 85.11627906976744%; position: relative; bottom: 0; left: 0; background-image: url('data:image/png;base64,iVBORw0KGgoAAAANSUhEUgAAABQAAAARCAYAAADdRIy+AAAACXBIWXMAABYlAAAWJQFJUiTwAAADgklEQVQ4y4VUS28cRRDeP8GNf8CVExdunBBIcEICBBfuPA4B5RCJCxJBkXgksYIiHkLYkbBIFFCw1yRRhOxdO2ucZf3cZB8zO7szPdMz/ZiemZ3YHzW9m0iRgjiUuqu76+uvq77q2oQJpOkxeJyCJxoRVwi5RBiRqRQsz8GEBgsTGkswBTBOfiSeMB4rDNwQNRYKrLevI+AuWtttuK4LFgTweYhg5xDi2gZk14EiYHnwB9TfP0CFDqROIaWAkhJCCBijMRoT4MD18Panz2Fl63swP0HAAkRBiCjXcC5eR+/5D8GWbiMDMbv0GvzTz0DebyCdnkArRa8jYBoVmQV0vAl+qV9Ao7OKVBtore2t2cMC/bVNbJxawKjxD3KcYPDnEvZ+/gxi3Eea5XR2DiiVBbeAYz9C7/4QzsCz1DU9RUlN4CnlUMCRHHGqbUDX9bHTpZTwhPa1ZaWVJEBJcwnXY6jxRKGcKhS5QiIkDD01KwiAbp6aEmVaUhAF0wWGAnWcwMxZKTOFKkBxRKDI4VGBa31X4OJSgis3OMKQY/3XGKvfxvCcEIubX2OxcQGCkp+lBlf3WviquYZh6KMoHkI2F6CuvQfudMC6ffTa+6jtHXG8+tYu3j+9D99nOP/RAc68vovOzgBvLLyIN8+/hIk/QcRCfHz1J7x8+SxavSMcnwDe5XfgnHoW7MFdNL/4EuvfXELNHUW4desIzc0+MZFo3DnE2m9ty3Ztq46Vxgo455gWBW5vt7BYXwWLOXLy/cN76G3UIWk/imP0Kce1iERclpSfTCFOBGLBKekBJVnDaKqkNCT2mKppkFBQSLKqKlvtJ4r2aD8RyqrDG0eoVV0x05K2Va4CK6sOaKpulT9OQFma47s75/DJ8rs4Gu0iNwWSJLaXKKvDqsrhDNCY1C4KMQOvwNJUz5lQK0YRJLE4c+UDvPL5CySdu8Q8JWaJJVGxfSzs/wKcgc66oGKRkWzGwQTdwYOZ/mSlV/V/gOIJwMoyY7A36OHmvRb1r8ZxeYwRG6J58Bd8NoFIxPzJjwDpVzHp0wBpjcYyL1Bvb+Ps78twiGGZT3HgdrDc/BF9r2fb9VEv2xxG9O3kuZn35OzJs/zNGOaZwdAbYXO3Yy/Mswx+OMH2/pZVREZ+FWfovDeJqt8mtD3okA2cyVNt6PhwRwz9x35A31xIc9/61XqFsd8d4V/seNz4IariWwAAAABJRU5ErkJggg=='); background-size: cover; display: block;\"\n    ></span>\n    <img\n        class=\"gatsby-resp-image-image\"\n        style=\"width:100%;height:100%;margin:0;vertical-align:middle;position:absolute;top:0;left:0;box-shadow:inset 0px 0px 0px 400px white;\"\n        alt=\"IMG 5\"\n        title=\"\"\n        src=\"/static/002e2cabf19bf7c3dd2b5c1d138f1f43/40fad/IMG_5.png\"\n        srcset=\"/static/002e2cabf19bf7c3dd2b5c1d138f1f43/707e9/IMG_5.png 148w,\n/static/002e2cabf19bf7c3dd2b5c1d138f1f43/649e0/IMG_5.png 295w,\n/static/002e2cabf19bf7c3dd2b5c1d138f1f43/40fad/IMG_5.png 590w,\n/static/002e2cabf19bf7c3dd2b5c1d138f1f43/b3fef/IMG_5.png 885w,\n/static/002e2cabf19bf7c3dd2b5c1d138f1f43/301c0/IMG_5.png 1180w,\n/static/002e2cabf19bf7c3dd2b5c1d138f1f43/7acd2/IMG_5.png 1290w\"\n        sizes=\"(max-width: 590px) 100vw, 590px\"\n      />\n  </span>\n  </a></p>\n<figcaption align = \"center\"><b><em><sub>Final architecture</em></b></figcaption>\n<figcaption align = \"center\"><em><sub>source: draw.io https://aws.amazon.com/architecture/icons/</em></figcaption>\n<br>\n<p>This is how all these services were brought together to implement the pipeline using AWS services. We are extracting data from the Data Warehouse to Glue jobs where the preprocessing happens. It then stores intermediate results (as a least effort solution) to S3. The Training job starts as the next step after the Glue job finishes. Keep in mind that the name should be unique within the region and account and it is a required field. In the Step Functions we also specify the S3 location for the output artifacts.</p>\n<p>The next step is to set up a schedule. For this purpose we created a Cloudwatch event that triggers a state machine and executes the pipeline.</p>\n<p>The last step in the pipeline is to load output data from S3 to a database that will be accessible through an API. This task is solved using Lambda. It is another AWS service that runs code based on the incoming request or event. In our case Lambda populates predictions from S3 to Redis. Redis is an in-memory structure store that not only supports a caching solution but also has the ability to function as a non-relational database. It will override any specified fields that already exist in the hash, or if it’s new, create it. It works well for our use case. Finally, we created an endpoint using FastAPI that fetches data from Redis and provides it to the other teams.</p>\n<p>To create all these services manually and deploy them would be time consuming and difficult. Instead we are using Terraform. It is software that creates infrastructure and allows you provision resources quickly and consistently and manage them throughout their lifecycle. By treating infrastructure as code we can avoid costly mistakes that can come from manual setup of services.</p>\n<p>Here is an example of Terraform instructions that manages schedules in CloudWatch and triggers a Step Functions.</p>\n<p><a\n    class=\"gatsby-resp-image-link\"\n    href=\"/static/f02efe920295efbeee767de440838fb9/35075/IMG_6.png\"\n    style=\"display: block\"\n    target=\"_blank\"\n    rel=\"noopener\"\n  >\n    <span\n    class=\"gatsby-resp-image-wrapper\"\n    style=\"position: relative; display: block;  max-width: 590px; margin-left: auto; margin-right: auto;\"\n  >\n    <span\n      class=\"gatsby-resp-image-background-image\"\n      style=\"padding-bottom: 39.00481540930979%; position: relative; bottom: 0; left: 0; background-image: url('data:image/png;base64,iVBORw0KGgoAAAANSUhEUgAAABQAAAAICAYAAAD5nd/tAAAACXBIWXMAABYlAAAWJQFJUiTwAAABW0lEQVQoz51R23KDIBTM/39gO+mYxAtKvKCgiFzE7ZFk2qe+lJmdPUdhz7JcGKuxLBLj/MAgM7TjJ/Ed/ZhByJzqB+RSoxVX4gbL2mOaK0INpTm0GeC9x3EcCZc8z9F1HWp+Q1Fd8Ww5pOqT0InzsNJPCFXAugV7DPhrJcGiKFCWBfp+wDD0JPoF3hbkZKDpAmaTWLcxOVMLp35KOL9tboaxMrkMwSbRi7UWzrmEczm/wBKcX7HZma5jElunk0MfDP3XSciH7V2rH+cXrTUdMEnohLEjbVxp44wQN3IxIUYaGBaq5WugOweoNNgmlxP2/S3IGCO7HnqVqJscrLmjKDNUjLi6oazooTqe+uz+QRkz6hlKllFdoRtqyv8BpdRLkHPKxRisq0HTcNR1g6piEIOAEAJKzpBSYRxHwkQRBHJ8kKMj8RFfHGP8zfDEvu/kNFCWlq7s8Z91vvI3DgJm2foWbQYAAAAASUVORK5CYII='); background-size: cover; display: block;\"\n    ></span>\n    <img\n        class=\"gatsby-resp-image-image\"\n        style=\"width:100%;height:100%;margin:0;vertical-align:middle;position:absolute;top:0;left:0;box-shadow:inset 0px 0px 0px 400px white;\"\n        alt=\"IMG 6\"\n        title=\"\"\n        src=\"/static/f02efe920295efbeee767de440838fb9/40fad/IMG_6.png\"\n        srcset=\"/static/f02efe920295efbeee767de440838fb9/707e9/IMG_6.png 148w,\n/static/f02efe920295efbeee767de440838fb9/649e0/IMG_6.png 295w,\n/static/f02efe920295efbeee767de440838fb9/40fad/IMG_6.png 590w,\n/static/f02efe920295efbeee767de440838fb9/b3fef/IMG_6.png 885w,\n/static/f02efe920295efbeee767de440838fb9/301c0/IMG_6.png 1180w,\n/static/f02efe920295efbeee767de440838fb9/35075/IMG_6.png 1246w\"\n        sizes=\"(max-width: 590px) 100vw, 590px\"\n      />\n  </span>\n  </a></p>\n<figcaption align = \"center\"><sub><b><em>Example of terraform instructions\n</em></b></figcaption>\n<br>\n<p>It’s very straightforward. We need to specify an event and rule, using the rule with a cron expression.</p>\n<p>There are two steps that should be executed during deployment.</p>\n<ol>\n<li>Terraform plan: Checking the execution plan for configuration before creating/changing infrastructure.</li>\n<li>Terraform apply: apply the changes.</li>\n</ol>\n<p>With Terraform set up, we have all the parts in place for our system.</p>\n<h2>Conclusion</h2>\n<p>This is only one example of how this pipeline could have been implemented. When it comes to creating your own pipeline, you will have different choices to make. Here are some of the lessons we learnt to help you when making those choices.</p>\n<ol>\n<li>\n<p>Step functions did an excellent job of combining services. It provides a lot of flexibility for creating different scenarios and logic flows. Overall a great choice for managing your pipeline.</p>\n</li>\n<li>\n<p>It is a best practice to limit access to AWS services. In AWS it is the rule that anything not explicitly allowed is implicitly denied. Each service should get its own IAM policy to handle this. This can make it tedious to set up, but is worth it for the security and peace of mind it provides.</p>\n</li>\n<li>\n<p>On the same topic of security, you should avoid putting sensitive data into S3. If you are using Jupyter notebook, be aware that it has access to the public internet by default. You should consider if this is really needed for your use case.</p>\n</li>\n<li>\n<p>It is worth understanding the basic concepts for VPC’s. We didn’t discuss them here, but they are an essential part for running many of the services. Also some services will need to be in the same VPC to access each other.</p>\n</li>\n</ol>\n<p>I hope you find this helpful, when you start to implement your own Machine learning pipelines.</p>","fields":{"slug":"/machine-learning-pipeline/","tags":["auto1","DataScience","Data","MachineLearning","Architecture"]}}},{"node":{"id":"d980aa96-d832-53e1-a6c2-0f3710beb58a","frontmatter":{"category":"Engineering","title":"Application Cockpit Architecture Overview","date":"2022-02-17","summary":"An overview of AUTO1 Application Cockpit architecture and its most notable features","thumbnail":{"relativePath":"pages/application-cockpit/architecture-intro/thumb.png","childImageSharp":{"resolutions":{"base64":"data:image/png;base64,iVBORw0KGgoAAAANSUhEUgAAABQAAAAUCAYAAACNiR0NAAAACXBIWXMAAAsTAAALEwEAmpwYAAADfUlEQVQ4y61UW2tcVRQ+OyAiiIIRVBRT6UPibmIcze2cOff7LZOZzORq0lorMiQxzjgdxQfFB6VFyJPSWKrRNE2wIW3BovlrHrrZy7X3MJGCjxn4OGuvy7e+tfY5oyjn/dN27hNj55ioP9wnMzePycjX98jQF/vkQvcuudA5IIMbe2Rg/WfywrU98lrzgLzePCSX2ifk3c5DMvrZAzK8cUJeXjsiz1TvEpLvk3MXqOjf741o3/1G3//mDvVvHNDxr+7Q4c5tOtb9hY53f6VvNG9RpXKDksYOHVz/ib5yZZeOb+/TydY9Otrcp8Mf/06fr2JO+CN9NtulinXr8ZPo9inzd0+fvHPzIbv47TF79csjNvj5IXupdche/BSxecTe7Dxgb3UesaHWI3ax/Se71P2LjV0/ZW+3/mZDzcfsuaUTpuR/MGXqw21Qr7Zgcm0TSqub+NyG91a2YHRhA7EFY40tKK+1wb5yHdRlzF1ugbrUgpnFFmhLbVAXEQttmJ5vw0i0BUpkWzwLPB6YOne1ae7pM2cIDJX7psotdZK7aEeuwV30W+oEt7UpbmmTEnZZYIp7xjRXoiSB+UYDstkK2K7Xg+Oe2Y7lgWFYkOWzsLL6AURJCkmagReE4MUBxHh2vQAckYtQgiCAWq0GYRiC67oQRREIn7Bdz4VwIwY7tiHAojRN0OdBks/1YnkEOTbyfR8cx5E1im3bkkw4+nYcx9IWPm/RB9M1ZXKC03hYnGaZjAtyz0d1ji3PAopIStMUTNOUDsuyzmwJ3QZd16Xqer0u1YgcERN5Av2zJIzSCsS4kzCKIUxyqeopuA6YlimJxDrkKhCeGB3FiIn665Iju54PPnYXCX7QC4gEUdwfXxALW6jsNxKjr1++Cg280KcIsYDHUcRRLsddcCTm1WqVY3eJLMs4JnLXcbjvudw0DG6aBrctiyMZxyZc18vcskwJRexPSO93Ft1E1/5OhA8J5OtRqTVQGY4Yp4gMLFvcbG+yIIxxLQEoWPAPKijwWSBBgR0LVCVt4ceCAlUVUV4vVj/pFrXlj4q5xctFbeVakVbqRZxXi3SuUVQW1opkdr5QNE0bx1sqCSBJqVwulyYmJqSNNyj9hmGgXyvp2rT06WW1VFaFbci4eBqGLmPn//eFrAOoZgA7/S9EDBVK4Kc1gJ8lwukBYz38dz53gf8CwA4PmL99534AAAAASUVORK5CYII=","width":325,"height":325,"src":"/static/b08a403a7e9534ad64f67f1ad61141df/b3029/thumb.png","srcSet":"/static/b08a403a7e9534ad64f67f1ad61141df/b3029/thumb.png 1x,\n/static/b08a403a7e9534ad64f67f1ad61141df/8d141/thumb.png 1.5x,\n/static/b08a403a7e9534ad64f67f1ad61141df/ee72c/thumb.png 2x,\n/static/b08a403a7e9534ad64f67f1ad61141df/5dfa8/thumb.png 3x"}}},"authorName":"Przemyslaw Walat","authorDescription":"Przemyslaw is an Expert Software Engineer based in our Szczecin office","authorAvatar":null,"headerImage":{"relativePath":"pages/application-cockpit/architecture-intro/1.png","childImageSharp":{"resolutions":{"base64":"data:image/png;base64,iVBORw0KGgoAAAANSUhEUgAAABQAAAAMCAYAAABiDJ37AAAACXBIWXMAAAsTAAALEwEAmpwYAAACaUlEQVQoz0VSS08TURSelQtXrkhI1JjYhQuIIZEC7bzuvTN3bjtDH5TSQlqUBGGlKOGZCOnWP0Di2oAJCTHEgH9O5/Oca6csvsy55/Gdc74zTnjx60/y7Teii/t87utPlEbXmD6+wtTBJab2r/DkwyUe737H9N41nn+6wbOPNyh9vsXM4R1mD+7xau8OT9/f4lH3Gk76A86b3ROUd0/z8s4pZreO8HrrBDObxyitH+Bl/xCl3hHmt85Q3R6h/PYLyptnmB+eY35wjsXhCAsD8m+MMLd2jhe1EziJknlmEqjAy4VbQRR4UH4VEUGHHmLhQrhLiMnWMoDwKggqCwiriwiqCxMIdxHSX4JjanV0VrvIlpcRCkkQCEIxsUNXwnV9JKaG7lrPfqM4hlQRpI4Q6wRCqnG+hBNTsN1uQ2sNKSWSJAH72JZKQu8kCHSIOIpRr9fJr5BmTahIQTcSNBpNRFEEQWRc4wiagkn4wRMxsTEGgqbkWDSM4SvfEmZZRsX0bbQQhiE1VDSttnkFnGRM4PsegsAnBGT7tsCiGsKtuig2iYikiHEeg2sKn2No/HqakRYaupaNO8mHrjS5T42YsFarQREh+/lbpzdvNJGIV5a0ApOx0JE25BQ2gRMZLAd3LgqZjAsVHaW/McDKSmcimSU0JsnHRbmgQkUCt1otewBGmqZ2GiZVdCQrC60pRIher2/JXNe1cV7d4SJaJS9W5Ek6nc5EE/Z5nke/i0G63LRy8CEUoYjz5flA/1eW8i8bFMg5gdfiaz4kk4aeiyRrY317H43uEI3VAVq9dzBpE9qkMFkLWbtv3/8ADCO9cUtyqcEAAAAASUVORK5CYII=","width":1280,"height":786,"src":"/static/477fbbb52f4a6b56a8627202336a0dce/26421/1.png","srcSet":"/static/477fbbb52f4a6b56a8627202336a0dce/26421/1.png 1x"}}}},"html":"<p>Hello fellow concerned developers, or, if you are not yet concerned, while it is not the crux of today’s entry, I hope\nthat by the end of it, you might get interested in becoming one and/or find out how you could do that on your own and\nwhat it actually means for us. Still, the focal point of this post is to fill you in on the architecture of Application\nCockpit (or “App Cockpit”, as we call it in private), the newest tool AUTO1 has brought in for its concerned developers.</p>\n<p><strong>The “why”</strong></p>\n<p>However, before dwelling on the “how” too much, let us pay homage to propriety and start with the ‘why’ of the whole\nmatter. The main reason for the emergence of the Cockpit stems from the plethora of tools AUTO1 has adopted to help us\ntrace, measure and monitor our services, not to mention alerting us when things go south. This opulence often means that\nif you are looking for a specific bit of information, whatever the reason, chances are the data is there somewhere, but\nyou need to make a significant effort finding out where to look for it first.</p>\n<p><a\n    class=\"gatsby-resp-image-link\"\n    href=\"/static/7913f56e8ec1a8e1392f735128cb79a1/ea190/2.png\"\n    style=\"display: block\"\n    target=\"_blank\"\n    rel=\"noopener\"\n  >\n    <span\n    class=\"gatsby-resp-image-wrapper\"\n    style=\"position: relative; display: block;  max-width: 590px; margin-left: auto; margin-right: auto;\"\n  >\n    <span\n      class=\"gatsby-resp-image-background-image\"\n      style=\"padding-bottom: 37.099494097807764%; position: relative; bottom: 0; left: 0; background-image: url('data:image/png;base64,iVBORw0KGgoAAAANSUhEUgAAABQAAAAHCAYAAAAIy204AAAACXBIWXMAAAsSAAALEgHS3X78AAABLklEQVQoz12P626CQBSE9xlUKO+g9YJGQRFhuUbFhED7/q/RNDnTnZPSH/1xwpyZ2W8XMw7D9ziOsDYXay04RVEgz3MdanrUUzb1mqZB27Yoy1KqqoKbLzN+fOLx7KQoK9zvd5p6mLrrOtX0Xq+XAggihB7h7P36wt2UbSf1Y0Ca0bC4XC44nU6o61qL1GmaKvh8PiOKIoURzOx2u6Hve8myjJmYrGiQla0Lj3+/xSILk06SRIeaQILjONb9er3qBQQzM0XzFFs/WNInTwE1h5ovpE8IYbyYIO68yOXC3P2dmP0xxiFK5H29xn4fYrvdYrlcYrfbYbPZqOY3DEOsVisd7mvXn3qEO7DQM54fiPcWYD6fi+d5WCwWmM1moPZ9XzU9juvo/M+CINCzZPwA2yvoGlsGHf4AAAAASUVORK5CYII='); background-size: cover; display: block;\"\n    ></span>\n    <img\n        class=\"gatsby-resp-image-image\"\n        style=\"width:100%;height:100%;margin:0;vertical-align:middle;position:absolute;top:0;left:0;box-shadow:inset 0px 0px 0px 400px white;\"\n        alt=\"2\"\n        title=\"\"\n        src=\"/static/7913f56e8ec1a8e1392f735128cb79a1/40fad/2.png\"\n        srcset=\"/static/7913f56e8ec1a8e1392f735128cb79a1/707e9/2.png 148w,\n/static/7913f56e8ec1a8e1392f735128cb79a1/649e0/2.png 295w,\n/static/7913f56e8ec1a8e1392f735128cb79a1/40fad/2.png 590w,\n/static/7913f56e8ec1a8e1392f735128cb79a1/b3fef/2.png 885w,\n/static/7913f56e8ec1a8e1392f735128cb79a1/301c0/2.png 1180w,\n/static/7913f56e8ec1a8e1392f735128cb79a1/ea190/2.png 1186w\"\n        sizes=\"(max-width: 590px) 100vw, 590px\"\n      />\n  </span>\n  </a></p>\n<p>Once you have figured it out, more often than not, you are invited to dance your way around Kibana, GitHub and Grafana,\ncorrelating your findings, perhaps with a touch of Consul and AWS Event Log to help you along the way, sprinkled with a\nChronograph query here and there to spice things up. Neither is it fast, nor easy, nor pretty. What’s worse, we have\nnoticed, that as our ecosystem grows, it gradually becomes harder to know what to look for and where to find it and the\nlearning curve for new hires has started growing ever steeper.</p>\n<p><strong>…. and the “what”</strong></p>\n<p>Enter the App Cockpit. In short, the Cockpit is meant as THE tool which lets concerned developers quickly find their\nbearings and gather relevant information pertaining to a particular service running in our ecosystem. Being concerned,\nactually, for us means that you care about your services and the ecosystem as a whole and you want to keep your\napplications in tip-top shape as well as you feel responsible for troubleshooting them when needed.</p>\n<p>Keeping that in mind, we’ve designed the App Cockpit to be the first place you check when you want to find out who owns\nthe service, what other services or resources it relies on, is it healthy or not and what its main software dependencies\nare. It is also meant to be the place you visit first when a service starts misbehaving to be presented with quick\ntroubleshooting aids, or, if you are the owner, to perhaps bask in glory when everything runs smoothly.</p>\n<p><a\n    class=\"gatsby-resp-image-link\"\n    href=\"/static/819dad8db10b5028167d822d8695e0db/a1751/3.png\"\n    style=\"display: block\"\n    target=\"_blank\"\n    rel=\"noopener\"\n  >\n    <span\n    class=\"gatsby-resp-image-wrapper\"\n    style=\"position: relative; display: block;  max-width: 590px; margin-left: auto; margin-right: auto;\"\n  >\n    <span\n      class=\"gatsby-resp-image-background-image\"\n      style=\"padding-bottom: 53.18118948824343%; position: relative; bottom: 0; left: 0; background-image: url('data:image/png;base64,iVBORw0KGgoAAAANSUhEUgAAABQAAAALCAYAAAB/Ca1DAAAACXBIWXMAAAsSAAALEgHS3X78AAAB8klEQVQoz1VSy47aMBTNBgYJ6I8gpFJAIkBC4tjOO6GESqxowqKfMMNuRuq0/zq7foFPr02CmMXRffjc63t9bHmeBy5CxblAEAQGvu+Dc27Q+YwxaG6H3W4H13U7qDb+Z+33e5UXBRG8R8IdO/dWaNs2lsvlHeaMmjxAbTYbWGEYQkqpWMDgBz4CFoC1k2rf574pqH/WuF6vuL5c8fzyDBlKM72pY77iAcd2u1WWJz1KMDjCAQsZmM8M0YAu4Ckz69a/Grz+ecX7+2+8/X1DkiRwmANxCCATAY972Gz1hHGIUErwkCOOYwgh9MQQGqFAksYm73Ja3Xex3thYbVeIoggiEoj2IeI8pqYSjuvAKrICUfYdaZpAxIJsijiJkWUZ+YQsN7mSbJkXyPMcUkgztXkesxFDuzIsMw0RHhXU0Ct7Pvnt+rqBhn4eX5+3nEcYUQpSuCToRKdc9y06XxNXq5VR2taWYpumMXGL9XptYB1/HEmxWOl31Ip3f0/7+j11w9PphMvlgrqu0ZBtKG6ormmaW65p1Pl8huM4pHJaqenXb/S3FmaKxWKB6XSK+XyO2WyGyWRimldVhcPhgOp4REUKH+iye45sWZZKc63hcIgv47EajUbQvrbj8fiTPxgM0Ov10O/3b/bpCf3POaUt8T/+A9K1gDWcRDsxAAAAAElFTkSuQmCC'); background-size: cover; display: block;\"\n    ></span>\n    <img\n        class=\"gatsby-resp-image-image\"\n        style=\"width:100%;height:100%;margin:0;vertical-align:middle;position:absolute;top:0;left:0;box-shadow:inset 0px 0px 0px 400px white;\"\n        alt=\"3\"\n        title=\"\"\n        src=\"/static/819dad8db10b5028167d822d8695e0db/40fad/3.png\"\n        srcset=\"/static/819dad8db10b5028167d822d8695e0db/707e9/3.png 148w,\n/static/819dad8db10b5028167d822d8695e0db/649e0/3.png 295w,\n/static/819dad8db10b5028167d822d8695e0db/40fad/3.png 590w,\n/static/819dad8db10b5028167d822d8695e0db/b3fef/3.png 885w,\n/static/819dad8db10b5028167d822d8695e0db/301c0/3.png 1180w,\n/static/819dad8db10b5028167d822d8695e0db/a1751/3.png 1446w\"\n        sizes=\"(max-width: 590px) 100vw, 590px\"\n      />\n  </span>\n  </a></p>\n<p>To achieve all that, the Cockpit needs to pull data from multiple sources, aggregate it and present it in a comfortable\nand meaningful manner to the users, not to mention being fast and efficient about it, which leads us to the heart of the\nmatter - the architecture of App Cockpit.</p>\n<p><strong>Finally, the “how”….</strong></p>\n<p>When designing App Cockpit, we were striving for what probably every software engineer worth his hide can recite even\nwhen woken up at 4 am in the morning - we wanted the architecture to be “open and easily extensible, yet simple and\nrobust at the same time\", preferably allowing polyglot extensions to let every team in the company contribute and share\nwith others if they came up with a cool feature. In order to achieve this, we have split the Cockpit into 5 logical\ncomponents distributed in 3 different layers, as depicted below, with the descriptions to follow.</p>\n<p><a\n    class=\"gatsby-resp-image-link\"\n    href=\"/static/a5fde0a614ee1d33cf9b3d2224330a72/b6c2c/diag-1.png\"\n    style=\"display: block\"\n    target=\"_blank\"\n    rel=\"noopener\"\n  >\n    <span\n    class=\"gatsby-resp-image-wrapper\"\n    style=\"position: relative; display: block;  max-width: 382px; margin-left: auto; margin-right: auto;\"\n  >\n    <span\n      class=\"gatsby-resp-image-background-image\"\n      style=\"padding-bottom: 178.79581151832463%; position: relative; bottom: 0; left: 0; background-image: url('data:image/png;base64,iVBORw0KGgoAAAANSUhEUgAAABQAAAAkCAYAAACJ8xqgAAAACXBIWXMAAAsSAAALEgHS3X78AAAG50lEQVRIx41WC1BU1xm+M2aySWY6iVUwMYFlQdP4qMbH8paXkIdEqVWCThIzTkybmsrE1pj4aGRZILEsAuG1PIoaiIbU2vpKUqNO2szIU4FtgsuCAcIUfAUFd/fu497z9T+XXVweyfTOfPOd+5/vfOc//3925wpB8RWqgBijKnRdrSpkZaVKHWtUqeOMquDEGtX9P1PPnaaaHjvt/oefve+BGZHTVI+s5GOKxT/kt9Q/YgfuC4gyPKCOK1d5IZChKSC2vDN0fa2ZDM0UNAfFlxNXXAmKK+sOiikY1MQW9QfHl13VxBX3BcXkX9fElfTSphZaSxrS+4AbQp1QjfAX6zAn6RCCaMyhWVmNgNgq+IWXwy9iPPwjy6GOH9V49V5wQ/vs6DJGGUqUoUwZyhSTn1hhlJet/Uje/O7n8itvn5E37fxM3kT8KvGG7afkJ5+pkgNjR7V8jRfcUIzacBSRaR+z4IRKUBBkjJ9ri7Blzz/BH5dbhluS4XJJyvvNITvCUz8GJQJNQgX4Kb0Q5iZVidv05xG98Qjjk9wwmAxnhhaDMkNX7218Y7mJrr7buPTNNXR030KzaRARaT9iOCexUnxTd44Mj04yfHnHGVy7aUU3mQ1cv4u+/w7j+4ERtHZcB5UDj68gw/gJhmQgvpV9AUmb/8q4GYfSKM/RF6w6qHBq+klEbTgCqh3mPV+tGHm14wxDKMO0t05x8ViG3knvBv4RJVi37QS062rwaGQpqBnjNh5vuLJS/N2+L7HC58i+hpxnaIuxY/9XWPPGcaUUvnOTDOkI4rbM8+My5DX0Cy/BesqK16uxfQCmzhtKUzi++HcPlqQcBl2tyTUkE/Gp56oRtr6WcSNv7aYvL8LWjC+Va2LpGVKaYekdwuANK+7anAhLrZ26y8HxRjEwpozuVQ2bm1jBNPFGhWdoP2TbdGcZ+bE7w3Y2cldkVquDyZLEXE4Xi3yxhj0RXcpCEsoZeYxB8I856Ji54iAWra3DY3GH4UfjWbGHMD3iL1iSegzbDY34/fsXkV/7LfaWXEL6/ga8rvsa6qQazIyuhn/MIcLBMQirUzJsyav3ya9uzpNS1mZKL6zZR8iQKC49l/yeFJu4VwqP2SX9dmuxnPLrLCkybrcUl7RXojXSatK9MAGCQx9oF/fNYo785bIjSyM7MgNkihEHyk5iV+ZjsutdlSx9+prsLomWXbselF36x8c0EyFYdUEOx+E02IviYM3UwKpTEweNgsb28lUQj2yGq74SzrPZEGtfga0gEtaMgFG9V+uBQEHReTwd9tJERuYENaMJ4kBmzZrD5KE+Rp1hTHIzxmSlSa7WT5l1z0xm1Qd71tyDYNOHiM6612EvjGY2T4Y2vltGIGzZT4I5rZBvWCD1X4bbfBZMckLqugDr7hmw6YMVrS+EkZx5or3uN7AWRLER/Rw2otOwkczgMXZcrGKu3kbmHuxgrr4W5vquntlP7GQjf5pN+pBRrQ8E6GY5oPMDDixi0M8GdP6EWR7Q+L1HgN0PAp9sAkqjafzQaMw7P6YdhXA6e6PtVFaadO7Dre4z77/kPp210U2xMZzKedl9Qpfq/lfVLumLvC3uk/o0ir00TuML4ZcGk7gg14QEYwdbnGdiCw3tjGLEJoWfPvAfNj/XxFI/srCo4m+VONf5ahb6QAgzNDq0uU1ILmtlEXlN0OY2gvPCnHq8WXcFA3dEXKU/2X762++9ZVO4rX8Yz5ZcxlL61dB6hObeg6A1NIlLc5vxfFkbC8trZstzmxSen93A0o91sv7bIrPcsLG+IZH13LIz8zWrgtXl7WzxB40s1NDMKKExCLRYfKOuE2sq2vkkyBAUw1PZ9djx9y7IdAvv2N2wOSUMi25YiQeHHUg2tmHxfsqK1mh9IEQcaBb3nP4OKT6GfGIJiSlr7Dl9FduPW1DdMADD+T68/Y8u/IHeo/JbsMyjnWDYIr5z8uq4DL2mS//chEUfNGKuvh5bjl7BM6Wt+EVWvRLzapZPMBXIRPwj7fqrShPTTjBcQobzcxrI8CKoZkqcGy6g2LIfM+Q13HmyGymV9zLkTAXnm6Dwq++x/1wvapsHUfJ1PwwX+pBztgcxhZeUE0w6cnRBi5h+zMKLzLt7rymUyTsnuuF9nPTlwBvEH94g3pSnp2oKzzD9b5ZxTZnK8AerC3bPp8hPGkbmt4gZn/dMuja8Tq8duYLW/hHU99xBc98wGnuHFT7X+QMSi1uVGmunyNAeX3SZrTK2yTSWyVABTcq0QJ6X3SBTY0bZA9pMXubRePVeCEo389qQXNmBiMJ2aPPbEFpAIA4raEd44dRQNFOAG5poNzNdYjNlaKZdzPRzVFiBwYPc/w//A/3PElylj3URAAAAAElFTkSuQmCC'); background-size: cover; display: block;\"\n    ></span>\n    <img\n        class=\"gatsby-resp-image-image\"\n        style=\"width:100%;height:100%;margin:0;vertical-align:middle;position:absolute;top:0;left:0;box-shadow:inset 0px 0px 0px 400px white;\"\n        alt=\"diag 1\"\n        title=\"\"\n        src=\"/static/a5fde0a614ee1d33cf9b3d2224330a72/b6c2c/diag-1.png\"\n        srcset=\"/static/a5fde0a614ee1d33cf9b3d2224330a72/a384c/diag-1.png 148w,\n/static/a5fde0a614ee1d33cf9b3d2224330a72/9ca5b/diag-1.png 295w,\n/static/a5fde0a614ee1d33cf9b3d2224330a72/b6c2c/diag-1.png 382w\"\n        sizes=\"(max-width: 382px) 100vw, 382px\"\n      />\n  </span>\n  </a></p>\n<ul>\n<li>\n<p><strong>Data Collector</strong> - the component responsible for fetching data from multiple external sources, parsing, aggregating,\ntransforming and sending it on its way to the data sink with a tiny stretch it might be called the ETL of App Cockpit</p>\n</li>\n<li>\n<p><strong>Data Sink</strong> - a simplistic component with a one-track mind, whose only job is to receive data in a well-defined\nformat from the Collector, occasionally perform some simple and generic operations on it (like basic validation and\nparent id resolution) and persisting it in the data storage</p>\n</li>\n<li>\n<p><strong>Data Storage</strong> - a simple storage that holds the current snapshot of App Cockpit data</p>\n</li>\n<li>\n<p><strong>Data Provider</strong> (aka Backend For Frontend) - exposes an interface to serve everything our UI needs, both serving the\ndata “as-is” from the storage as well as applying some additional logic to further aggregate and transform it for\nspecial purposes</p>\n</li>\n<li>\n<p><strong>UI</strong> - yer plain old Front-end - by design contains no logic, it only displays the data as provided</p>\n</li>\n</ul>\n<p><strong>Sweet, but how about a model?</strong></p>\n<p>The specifics of the model run outside of our scope, however, getting it wrong could severely hobble the architecture,\nso it is a story in and of itself, which will have to wait its turn to be told in another blog entry. For our purposes,\nwe will only briefly touch upon it, mainly to establish the domain.</p>\n<p>Truth be told, App Cockpit is pretty much a simpleton, since its world consists of only two kinds of things: Readings\nand Resources and everything it does revolves around the pair.</p>\n<p><a\n    class=\"gatsby-resp-image-link\"\n    href=\"/static/620475cb60f40b22eca2eb0e91489dcd/bdb41/4.png\"\n    style=\"display: block\"\n    target=\"_blank\"\n    rel=\"noopener\"\n  >\n    <span\n    class=\"gatsby-resp-image-wrapper\"\n    style=\"position: relative; display: block;  max-width: 590px; margin-left: auto; margin-right: auto;\"\n  >\n    <span\n      class=\"gatsby-resp-image-background-image\"\n      style=\"padding-bottom: 60.852459016393446%; position: relative; bottom: 0; left: 0; background-image: url('data:image/png;base64,iVBORw0KGgoAAAANSUhEUgAAABQAAAAMCAYAAABiDJ37AAAACXBIWXMAAAsSAAALEgHS3X78AAABtUlEQVQoz3VSbZabMAzkGkl+5ZVDYMDG+AvsBBfSvf9F+mpVEssub7v9MRGRRiONoLrf77/ruob6R11utxscuF6vcLlcOJ7xb4655brn/1TBB1jyUkKawBoLxhiw1sI8z5BSAu89OGc5hhA4RzXnHPMs8qdphpieRY8jkGDZXhv44EvOmUnc+HggaUd6ZliWhUVJ8PV68WDKEbCn/Hp7g2EYSqWUYhEpJU8l4oiTDG4rlQY1aNDGgdaaOVQj4aOPeiQ+O+9AYb2i9WnKuq5shYg4iaE5qncMHzhzjmeKNLAiC9u2QUQxEqTpZ0zTxCdYlifWJ97oK4dAveSCLcfHAi7M0Pc9TyFQvus6vGViUa1HeOQNBowH5wBtmH9m7qmk7GH0CbSbQTQNCCF2tAIa/D9aw1u1KO7iCqOLn5wPbgshTrtl+jEBBU3AYgMtFg+IRoD1lg/e9/jSYgaJL+nMYeCwOeH9JW8oCwmOPpYWp/1PsMNz+LTCYDzWxDeCcRdU+4bFoG3xRZAskyAdvet6sHNGYclOzpaJG+awW1ZKltF6MH4qdNQz+Bx4Q/o2z5/Hd6Aa8f8CyFeeSMaQMngAAAAASUVORK5CYII='); background-size: cover; display: block;\"\n    ></span>\n    <img\n        class=\"gatsby-resp-image-image\"\n        style=\"width:100%;height:100%;margin:0;vertical-align:middle;position:absolute;top:0;left:0;box-shadow:inset 0px 0px 0px 400px white;\"\n        alt=\"4\"\n        title=\"\"\n        src=\"/static/620475cb60f40b22eca2eb0e91489dcd/40fad/4.png\"\n        srcset=\"/static/620475cb60f40b22eca2eb0e91489dcd/707e9/4.png 148w,\n/static/620475cb60f40b22eca2eb0e91489dcd/649e0/4.png 295w,\n/static/620475cb60f40b22eca2eb0e91489dcd/40fad/4.png 590w,\n/static/620475cb60f40b22eca2eb0e91489dcd/b3fef/4.png 885w,\n/static/620475cb60f40b22eca2eb0e91489dcd/301c0/4.png 1180w,\n/static/620475cb60f40b22eca2eb0e91489dcd/bdb41/4.png 1525w\"\n        sizes=\"(max-width: 590px) 100vw, 590px\"\n      />\n  </span>\n  </a></p>\n<p>Accordingly, a Reading is a piece of data that describes a Resource in some way, e.g. a metric value, service name,\nowning team, the version of Docker image it currently runs etc.</p>\n<p>In complement, a Resource is a concrete entity for which Readings can be collected and then served; it can be a physical\nentity, e.g. an instance of a service or a logical one (e.g. a definition of a service, e.g. “authentication-service”, a\nsoftware dependency e.g. ‘log4j”, etc). In addition, Resources can form hierarchies, since they support a\nparent-children relationship.</p>\n<p>Given all the above, the main purpose of the App Cockpit is to collect Readings for tracked Resources and present them\nto the users.</p>\n<p><strong>Getting physical.</strong></p>\n<p>Having established all the basics, we can now shift our focus to the physical application of the architecture. We will\ngo over logical components one by one and discuss how we have decided to implement it, to finally sum it all up with a\nfull diagram illustrating the complete solution.</p>\n<p><strong>UI</strong> - we have chosen to implement it using a customized version of Backstage app, built using React JS and available\nat <a href=\"https://backstage.io/\">backstage.io</a>. Since it provides many handy features readily out of the box, it has greatly\nstreamlined the UI development process. Provides a web interface for the users, communicates with Data Provider over\nHTTP via REST endpoints.</p>\n<p><strong>Data Provider</strong> - a simple, easily scalable service built in Java using Spring Boot, exposes REST endpoints for UI\nwhile fetching data from the datasource over JDBC; Apart from serving Readings and Resources it also has a concept of\n“calculations”, which come into play when returning plain entities is not enough to present the user with the desired\ndata. Calculations encapsulate any form of additional aggregation, correlation and transformation logic that is\nperformed on the data in order to enable the UI to present more complex views without placing actual logic in the UI\nitself.</p>\n<p><strong>Data Storage</strong> - nothing fancy there, while we have briefly considered using a temporal db, we have decided to fall\nback to plain old PostgreSQL, since the Cockpit is operating on relational data; the decision has paid off, though,\nsince we were able to leverage the built-in JSONB type to store heterogeneous Reading data in a single column. It is a\nsimple, yet powerful mechanism, which makes the system extremely pluggable. Since a Reading is actually a free-form JSON\nobject, all throughout the ecosystem, the only components that need to know its exact model are the Collector\nimplementation that is going to publish it and the actual UI component that is going to display it. Every other piece of\nthe system can be safely kept in the dark and will be more than happy about it.</p>\n<p><strong>Data Sink</strong> - implemented as a simple lambda function, it has a well-defined job to do; it listens to SNS (AWS'\nmessaging product) events published by the Collectors to its proprietary topic, extracts the payloads, performs simple\nvalidation logic and persists them in the storage. In addition, it is also responsible for properly resolving parent\nResources for incoming children. This piece of logic would fit in the Collectors as well, however since we have decided\nto make them stateless, it has inevitably found its way to the Data Sink.</p>\n<p><strong>Data Collector</strong> - it is the only component for which the logical diagram differs from the physical one, albeit by\ndesign. This stems from the fact that the Collector has been meant as the main extension point that other teams,\npossibly using disparate stacks, can easily plug into when adding new features. By all means, a Collector, as previously\nstated, is any component that fetches data from an arbitrary source, transforms it into Resources and/or Readings and\npushes it to the Data Sink SNS topic.</p>\n<p>In consequence, when you want to start collecting a new type of data or start collecting it from a different source, you\nhave two options - either add your implementation in one of already existing collector services, or write your own new\none, and the only requirement we superimpose on you is that when you publish to the Data Sink, you follow the format we\nhave specified. At the time this post is being written, we have 3 distinct Data Collector implementations:</p>\n<ul>\n<li>\n<p><strong>Main Collector</strong>  - a general-purpose collector, written in Java and providing data for cross-platform features,\ne.g.:</p>\n<ul>\n<li>\n<p>service ownership</p>\n</li>\n<li>\n<p>incidents</p>\n</li>\n<li>\n<p>Docker images used and their vulnerabilities</p>\n</li>\n<li>\n<p>service health and deployments</p>\n</li>\n<li>\n<p>currently running instances</p>\n</li>\n<li>\n<p>code insights (quality, PRs, contributors etc)</p>\n</li>\n</ul>\n</li>\n</ul>\n<ul>\n<li>\n<p><strong>Java Collector</strong> - yet another service written in Java, as the name suggests it collects data strictly related to\nservices written in Java, e.g.:</p>\n<ul>\n<li>\n<p>runtime information like JVM and GC being used</p>\n</li>\n<li>\n<p>selective list of JVM parameters passed to the instances</p>\n</li>\n<li>\n<p>egress dependencies for Java services (in this context, the upstream services)</p>\n</li>\n</ul>\n</li>\n<li>\n<p><strong>PHP/Golang collector</strong> - the first contribution coming from outside the original App Cockpit team, written in\nGolang, it is going to provision data for PHP and Golang services, e.g.:</p>\n<ul>\n<li>\n<p>egress dependencies</p>\n</li>\n<li>\n<p>vulnerabilities</p>\n</li>\n<li>\n<p>possibly many more to come</p>\n</li>\n</ul>\n</li>\n</ul>\n<p>With a picture being worth a thousand words, let’s have a look at a diagram that gathers all of this in one place:</p>\n<p><br><a\n    class=\"gatsby-resp-image-link\"\n    href=\"/static/f576711fa336eca6db48f2c1cb19c47c/b2341/diag-2.png\"\n    style=\"display: block\"\n    target=\"_blank\"\n    rel=\"noopener\"\n  >\n    <span\n    class=\"gatsby-resp-image-wrapper\"\n    style=\"position: relative; display: block;  max-width: 531px; margin-left: auto; margin-right: auto;\"\n  >\n    <span\n      class=\"gatsby-resp-image-background-image\"\n      style=\"padding-bottom: 161.5819209039548%; position: relative; bottom: 0; left: 0; background-image: url('data:image/png;base64,iVBORw0KGgoAAAANSUhEUgAAABQAAAAgCAYAAAASYli2AAAACXBIWXMAAAsSAAALEgHS3X78AAAFzklEQVRIx4WUi1PUVRTH9x9oemmpZSCogA+QeCUo78U1RVI0C8lHZaBRzvgaDQRDHuKykAgBuzxCwdGE1TEtk9GymkQRJQ3IUHuwNpGZ5ANFw0/33mWJ3ch25ju/e+8553u/53vvXY27rpyoxTuZvayW2KQ6guOqcQ4zMuWlHcx7q47Zy2uZlViLW1QZTqFGFq75iPW5R1mdXc96/VESUj5mjNbEqHATntEfoBmjNfL58Xau3+jm5s3bZBZ+hsY5h9T8o/z2+3XaLvzG6dNnGacz8ohvIY3NPyJ/PT13uXfvrhjdJ23rFzzis40JMyoloYm5SbUkpn7C6qxDpL9Xz/wVe0k21LMq8xCvrjtA7LKdjI40MiSgiPyKr+m63s0Plj+51NHFjVs9lO9p5uFnC5g4UygcHVnGE5Pf53H/IsLjq+kUqq5eu8XPHVeUFY/6FfFkYImywSm0lJHBJSxL/ZTkvGOszDqivpGLdjM8qJjxzwuFo7VlKtlDVyF2qCAm8UNefHsvs5buYlJ0OWOVd6WMCisVeaXKxxFTShgZUsrTgvypqSXq+4yYj5tegcY10qQmstBtWrlKlAaPDDHiHFnFhJnb1WZPhVTiHF6GS7hRtG9S+S4RJiVmlFgbaSOUJFKBLSCTvWZV4SnUhsbmEDinGC+hNGJuNt7RpUyMrlI5thob+hXqXqtV7DaPZGvaJWaiFlZSuzmMFUkr8dZtpn7rFBLezmF6wkHcp5UpgkEJoxPNPBM6gFCoHKO13jnvmUW4i3adIrbjO6sIl7BikWdSrToNqHmwQkHoqgpKGBFcSfSCTBa8ns7ToZVqY0nmEv6AluVldFQoCUeFlTA0sJJ1SXHkJ7/MiKkmdcqKMOIBhI7ybQcjE2RsxusfMk9cfNm6XHP9P0J5se0Iw6weykIP8c4nzdmFZ+wePKZbb4OMyU0fQGi/mxxLuISXMiTQyJL4LNLfTCUobjczltah7q3DgdgRyrfsLA5AXpeBcA0vYVhwOWuXplG6Zi1eL9QQNL960FwJp5AS8fTK0ThFVIrTLBMvYQBCyxkmDsHrhZ2EL/0E7fJ6JsbUiEMyiliFivcjpFzVDBf5Y3RVaCZPN+A7zYDfv5CHb2QOujl6YuYbCIjKsa5rxVe75Z9vlB4/XR4+UbkExxSgsWT405HiYY8N4+lYPxpL7jQ6m+rpPPsVlqIXsSS7YckKxJI52YpN/lg2TqLjnbF0JLtzOd0HzZ2K2XSnO9G9yfkfZLhyK2UIPXXLobdH/aHePZTKne3zud/ZQm/7EXovHqP38hnu/97OnR1x3NowlO4tE9CQ7wVpj8G7Q2HjECvefQI2PAzb/KG5GlrrwBgBencwvwG7FsDueNibAB+tgMIASHkIsp3R7MtejDlrEGQvYV/GAvbnJrA/7001NmfEY06LxbxxnhVynDob8yYRy1rEfn0CGs/8VjxyWxhnEMjtgxi76b/Fr/A8c2t+4JXdPxFQdJ6xehHL+06grQ/fWWFoxV3k+xS0oQnKO4G/voEAB0w2nCCioJEY4xliy5rRbjtFoFgLGCRXwm9LA6FbG9E8Z2jEV38Sv1wr/MV8QnYD8dtbuHKjh7Zfb3Lulxvc7PmL1fva8cg8ToDIseXb4CM4grc2oZlTdlZN/HIbFRRhVgOLq1vV6Uqirtv31Hit+bwgbOgjtEc/oSz02XLSLvismIcWNLFqbzspBy6S+vElVu5pI2pbE94Ouf8ifPmDbxWBLeDbp3ZSzgmlZF7FOdX+c3mn8MoRlvTl+OpPDk4YV2VPOJBU+htZeBpdcTNB+afsSP6TULbmGJTKxgsfUw9e5HDbVY5+/wfZh3/Ec/MJO7KBdf2EAYbGQQnHCfMLjnXw9aUuvrhwjeIvLUQItQt3tCjVjr73E0pvbJ4M9EaaH2P6hv3nriiFr9a0qKKwgtN2PtrypW1TJWGgIFTXxQFSpWxRHsoioSoo3947x3y5HiII/wbp3Q12i8R3IAAAAABJRU5ErkJggg=='); background-size: cover; display: block;\"\n    ></span>\n    <img\n        class=\"gatsby-resp-image-image\"\n        style=\"width:100%;height:100%;margin:0;vertical-align:middle;position:absolute;top:0;left:0;box-shadow:inset 0px 0px 0px 400px white;\"\n        alt=\"diag 2\"\n        title=\"\"\n        src=\"/static/f576711fa336eca6db48f2c1cb19c47c/b2341/diag-2.png\"\n        srcset=\"/static/f576711fa336eca6db48f2c1cb19c47c/85b06/diag-2.png 148w,\n/static/f576711fa336eca6db48f2c1cb19c47c/75500/diag-2.png 295w,\n/static/f576711fa336eca6db48f2c1cb19c47c/b2341/diag-2.png 531w\"\n        sizes=\"(max-width: 531px) 100vw, 531px\"\n      />\n  </span>\n  </a><br></p>\n<p>Obviously, the actual process of data collection, as it is currently implemented, is a whole story in and of itself,\ndeserving yet another dedicated blog post, especially since we have elected to make all the collectors stateless, which\nhad its numerous consequences and challenges so keep your eye out for the next entry which is bound to come shortly.</p>\n<p><strong>Closing words and … we will be back</strong></p>\n<p>Having said, or rather written all the above, we sincerely hope that we have managed to pique your interest. Not only in\nour solution but also in actually getting “concerned” about your ecosystem, because it does not necessarily have to be\nboring, especially when you have the right tools to make your life easier. On top of that, you can have your fun\nbuilding the actual tools, deftly combining business with pleasure for a change of the usual pace.</p>\n<p>As hinted numerous times, this is only the preliminary article, meant to lay the groundwork for future additions\nexploring individual pieces in more depth… We are already looking forward to writing those, so keep your eye out!</p>","fields":{"slug":"/application-cockpit/architecture-intro/","tags":["auto1","Engineering","Architecture","Cockpit"]}}}]}},"pageContext":{"slug":"/tags/Architecture","tag":"Architecture","categories":["Architecture","Coding","DevOps","Engineering","ProjectManagement","QA","Social","TechRadar"]}}