![](\"https://lh5.googleusercontent.com/v1PbZaRZQJjKJbBQiQZOXRUGjLK_c000j3edWpNb7jhim2dDHl21ircQVV4-aabJnHA57-A68J7WXfTkMFFxeVro0kGAlSMh2PKZ72yW5IezNvN-FiM4Xm-siC7PhAS8ZarnG4imewzKJhjKxAR7Mt0A3R5BOkpSCDYW14ldqG2CLpW5XC-jbwJtvg\")
As you may have read about Application Cockpit in one of our previous articles, it is “The tool which lets concerned developers quickly find their bearings and gather relevant information pertaining to a particular service running in our ecosystem”.
\nOver time, AUTO1 engineers and managers grew fond of using Application Cockpit, and started relying on it on a daily basis. The increase in popularity, naturally, led to development of new features, out of which, one of the arguably most impactful ones was the \"dynamic service search\" (or \"App Cockpit Search\", as we call it internally).
\nAUTO1 platform currently consists of 600+ microservices, and on average two new services are created every week. Many times in the past, we needed to search our AUTO1 service inventory for services which were matching some specific criteria, e.g. are written in a specific language, using certain frameworks or libraries, or have certain features enabled.
\nAs it happens, this is actually where App Cockpit App Cockpit Search chimes in. From the user’s perspective, it is an UI that lets the user input some search criteria using the Cockpit Query Language (or CQL for short) and be presented with a list of services matching the criteria. One typical use case is searching the AUTO1 service inventory for the usage of a specific library in a specific version.
\nBelow is an example search for services using org.springframework.boot:spring-boot-starter dependency in version 2.3.0 onwards:
Once we were certain of what we wanted to achieve, i.e.: search our AUTO1 service inventory using dynamically defined criteria, we began the efforts of defining the architecture.
\nAs you may recall from our previous article about the Application Cockpit, we use Backstage as our Front-end. Backstage is fetching data from our backend-for-frontend service called “Application Cockpit Data Provider”, which in turn, is backed by a PostgreSQL DB.
\nAs Backstage itself comes with search functionality that could be potentially used for our purposes, we were left with two options, either to reuse the already provided solution and tailor it to our needs or create a solution from scratch.
\nAfter evaluating the Backstage search, which at that time, was still in the early alpha stage, it was clear to us that it would be a bumpy ride. As it is a full-blown search mechanism, or to put it more bluntly, an over-blown mechanism, compared to what we really needed:
\nAs we already use Spring Data JPA as the persistence layer to communicate with PostgreSQL in our data provider service, we have limited our options to two choices: dynamically built queries on top of joined database tables or database views. Either of them seemed to us, initially, as a good candidate for dealing with the complexity that stems from the heterogeneous data structures.
\nThe idea behind this solution was to create a set of SQL queries, backed by corresponding Spring Data JPA repositories, one for each attribute in question, execute these queries against the DB, and then combine the results on the Java side performing the necessary filtering and aggregation.
\nThe advantage of this solution was that we would keep things small and isolated. Each query is in a separate repository. Each one would deal just with the data structures it's aware of. On the other hand, this solution would come with a significant performance penalty, due to the substantial amount of data that would be transferred from the DB, just to be later filtered out on the application side.
\nThe alternative solution would be to dynamically build a large SQL query that would perform both the retrieval of the data and the filtering and aggregation of its results.
\nHowever, this approach would come at an increased maintenance cost. As every data structure change would require a corresponding Java code change, hence we would lose the benefits of having the data values stored in the JSON objects. On top of that, maintaining a large SQL query would quickly become a problem on its own.
\nKnowing the shortcomings of the dynamically built queries, we decided to evaluate another promising approach - the database views, that could hide the complexity of dealing with the heterogeneous data structures:
\nIn that approach, we would create a database view that could look like the following (based on our previous example):
\nwhich returns:
\nThe main database view, here called document_view, would be the aggregation of multiple sub-views. Each sub-view would only know about its data structures, hence we would achieve the desired single responsibility and encapsulation traits, making the solution more extendible and more maintainable. Even if something would be changed within one of the sub-views, it would not affect the aggregated view or other sub-views.
As previously mentioned, since we use Spring Data JPA extensively, the database view would be reflected in the code as a JPA entity which would allow us also to use the Criteria API to build a generic query on top of that main view and utilize the built-in PostgreSQL functions, that would come especially handy when interacting with the JSON objects.
\nAs the DB views seemed like a good solution, we still needed to answer one more question: whether to materialize them or not. As materialized views would be, obviously, faster to query, but that would not come for free, as we would need to build a mechanism to frequently refresh the materialized views, which would also induce a performance overhead. In the end, as we expect the App Cockpit Search to happen fairly infrequently and we value accuracy over query execution time, we’ve decided to not materialize the views.
\nFinally, we decided to move forward with the DB view based solution. Although it’s not a silver bullet and requires us to keep an eye on the query performance, as over time the number of sub-views will grow, which might impact the overall search performance, we consider this to be a fair trade-off for the gained extensibility and maintainability of the search functionality.
","fields":{"slug":"/harmonizing-application-cockpit-data-using-db-views/","tags":["Cockpit","PostgreSQL"]}}},{"node":{"id":"cfee7911-e765-5a6e-ba90-1eb2ce96d837","frontmatter":{"category":"Architecture","title":"Application Cockpit Model Outline","date":"2022-10-28","summary":"A short trip around the AUTO1 Application Cockpit data model","thumbnail":null,"authorName":"Przemysław Walat","authorDescription":"Przemysław is a Expert Software Engineer based in our Szczecin office","authorAvatar":{"relativePath":"pages/application-cockpit-model-outline/avatar_Walat.jpeg","childImageSharp":{"resolutions":{"base64":"data:image/jpeg;base64,/9j/2wBDABALDA4MChAODQ4SERATGCgaGBYWGDEjJR0oOjM9PDkzODdASFxOQERXRTc4UG1RV19iZ2hnPk1xeXBkeFxlZ2P/2wBDARESEhgVGC8aGi9jQjhCY2NjY2NjY2NjY2NjY2NjY2NjY2NjY2NjY2NjY2NjY2NjY2NjY2NjY2NjY2NjY2NjY2P/wgARCAAUABQDASIAAhEBAxEB/8QAGAABAQEBAQAAAAAAAAAAAAAAAAQDAQX/xAAWAQEBAQAAAAAAAAAAAAAAAAABAAP/2gAMAwEAAhADEAAAAYNcbctZHqE5AEDf/8QAGRAAAwEBAQAAAAAAAAAAAAAAAQIDEhEA/9oACAEBAAEFAnyRJdo0aBmiW9jE1khCKBOxPVY8/8QAGBEAAgMAAAAAAAAAAAAAAAAAABEBECH/2gAIAQMBAT8Beji//8QAFBEBAAAAAAAAAAAAAAAAAAAAIP/aAAgBAgEBPwEf/8QAHRAAAgIBBQAAAAAAAAAAAAAAAAERQQIgISIxgf/aAAgBAQAGPwJvpqi/CMJg4KTFN7l6P//EABsQAAMBAAMBAAAAAAAAAAAAAAABESExQWFR/9oACAEBAAE/IYU8XDGNg3v6Gar2x1WU9pKgnlpCG63tFjubo1NwXmn/2gAMAwEAAgADAAAAENfwvP/EABgRAAMBAQAAAAAAAAAAAAAAAAABEUFx/9oACAEDAQE/EHENHQ0h/8QAGREBAAIDAAAAAAAAAAAAAAAAAQAhEBFh/9oACAECAQE/EANUy+Z//8QAHRABAAIDAAMBAAAAAAAAAAAAAQARMUFRIWFxgf/aAAgBAQABPxB5rcKC843EOZKEWYB/dSvRVSN3XnMCCmUgo98hrV7b0d+xivco2sPKSmmz7hteDByVo6Gf/9k=","width":50,"height":50,"src":"/static/e4e77cd92050dbb448a088991f2d0c50/d2d31/avatar_Walat.jpeg","srcSet":"/static/e4e77cd92050dbb448a088991f2d0c50/d2d31/avatar_Walat.jpeg 1x,\n/static/e4e77cd92050dbb448a088991f2d0c50/0b804/avatar_Walat.jpeg 1.5x,\n/static/e4e77cd92050dbb448a088991f2d0c50/753c3/avatar_Walat.jpeg 2x,\n/static/e4e77cd92050dbb448a088991f2d0c50/31ca8/avatar_Walat.jpeg 3x"}}},"headerImage":null},"html":"Welcome, fellow concerned developers. In our previous installment, we have promised to delve deeper into the model that Application Cockpit is using underneath to see how it synergizes with the high-level architecture in order to make the whole product simple yet powerful and extensible. Be advised, that while we have always aimed for simplicity, our initial take on the model was a little bit more complicated, but after more-than-a-few heated debacles, we have stripped it down even more and so far we are pretty happy with it, despite more than a few \"I wish we had the initial design in place\" moments of weakness.
\nIn honor of the early version, we are also going to take a short trip down the memory lane to see the trade-offs and compromises we have decided to make along the way and features we have decided to add. We are going to discuss the direction we took and what were our reasons to do so (or in other words, \"what were they thinking ?\"). But for now...
\nAs we hinted in the previous blog post, Application Cockpit is actually binary, in a sense that its primary domain consists of only two data types: Resources and Readings and the (extremely) short story is that a Resource is an entity for which we are going to track Readings, while a Reading is a piece of information pertaining to a given Resource. The longer story follows below.
\nLike mentioned above, a Resource is mostly something that we are going to collect readings for, it can be a physical entity, or a purely logical one. Examples of currently tracked Resources include:
\n(semi)physical
\nLambdaDeployment
\nRdsInstance
\nServiceDeployment
\nServiceInstance
\nlogical
\nLambda
\nService
\nSoftwareDependency
\nSoftwareDependencyVersion
\nThe Resource has a very rigid model, comprising of only a few, well-defined properties, that is:
\nid - the unique UUID
\ntype - the type of the resource, see above for examples
\nname - a textual identifier, unique across one type; needs to be deterministically derived from resource's properties available at runtime (further details in tradeoffs chapter)
\nparent_id - id of the parent resource (if any)
\ncreated_on - a timestamp and a late-comer to the party, but we find it useful in many ways, especially during troubleshooting
\nIn order to publish (a.k.a create) a Resource, the collector will provide only the name, type and optionally the parent identifier.
\nWhen it comes to type, it is a free-form String. We have no predefined enumeration to represent it and the collectors (see previous article to learn more about them) are free to push whatever types they want, akin to the front-end which is allowed to ask for any type it wishes.
\nIn addition, the inclusion of parent id allows us to build hierarchies, so that our data closely follows the domain that we are modeling.
\nGiven the above, you should now have a general idea of what a Resource is and how we use it, so we can move on to Readings. Since we have established that it is generally a piece of data pertaining to a Resource, let's have a look at its properties:
\nid - the serial identifier;
\ntype - type of the Reading; free form String, akin to Resource type
\nresource_id - uuid of the Resource this Reading describes
\nvalue - free-form json document to hold the data of the Reading
\nstoragemode - determines the behavior when multiple Readings are published with same resourceid and type:
\nSNAPSHOT - only the most recent Reading is preserved
\nTIME_SERIES - all of the published Readings are stored
\nSince the values we store for Reading are free-form JSON, it gives developers a lot of flexibility in how to handle their flows. We store both trivial things, e.g.:
\nTo the humongous too-big-to-fit-in-the-picture guys:
\nAgain, akin to Resource type, the Reading's type and value are completely opaque to most of the Application Cockpit infrastructure, so individual contributors are free to ship anything they wish in those properties. The only components which need to be aware of the exact structure are the Collector that is originally publishing the data and the UI component which is querying the Data Provider in order to render the respective view. That way, many developers are free to add their own custom extensions, without stepping on each other's toes or needing to seek approval or assistance from the core Application Cockpit team, which in addition can remain blissfully unaware of the atrocities that the others may commit and attempt to hide within their custom Readings.
\nHaving dealt with the detailed model description, we can now move on to discuss how our journey with its design looked like. Of course we, being the pragmatic, more-than-once-bitten little members of the software engineering club that we are, were under no delusions about what is going to happen to our nifty design when it gets into its first bar fight against the necessities of real life. The only question was when we are going to be faced with our first moral/design dilemma and how much are we going to be willing to compromise our collective conscience.
\nWhen it comes to the little trade-offs, actually, the more observant reader might have noticed by now that something is a bit amiss in our picture:
\nIn this article we have learned that in order to publish a Reading for a Resource, the Collector probably needs to provide the resource_id to bind it to the right entity
\nIn the previous article we have mentioned, that Collectors are generally stateless and have no access to the storage holding Resource data
\nIn consequence, how can a Collector find out the correct id to publish the reading for ?
\nWell, it would that it could, but it can’t, so it shan’t. It’s quite a bind we find ourselves in, right ? We were adamant about not allowing collectors access to our DB, so we had to find another way to make things happen… A short look at the ReadingPublishMessage model, might provide some hints, though, so let’s start with that:
\nDoes anything stand out to you ? It is almost a reflection of a Reading, but with a slight twist, and you probably got it right, so enter Resource's name - that tiny little property looks like it has been put there in order to provide a human-friendly name for a Resource, but it couldn't be further from the truth than that. In fact, initially it was never there (we use Readings to store human-friendly names of the Resources anyway) and we added it specifically to solve the issue at hand.
\nIf you recall, the description says that it needs to be unique and possible to deterministically assemble from the data available in the runtime. Leveraging that, the Collector doesn't need to know the right id, it just needs to provide the correct textual name. The good news is that we have a way of overcoming the initial obstacle, but there is also bad news... This forces every collector to be aware of the logic required to construct names of the Resources it is interested in, moreover, the logic needs to be in multiple places and it needs to be synchronized. But let's leave it at that, since data ingestion is going to be the focal point of the next article in series.
\nAt this point a meticulous reader with a penchant for domain modeling might begin to wonder:
\n\"All right, a Resource can have a parent... That's nice, but is that really all that there is? You say you want to closely match the domain, but not everything is a child of something, sometimes you might want a Resource to 'own' Resources, so... how about that, smarty pants ?\"
\nThis was also something we were intensively brainstorming about and lots and lots of coffee cups later (and probably more than a few beer mugs, too) we have arrived at something we were not very uncomfortable with. Our main priority was to avoid complicating the model (remember ? we want people to integrate easily, so we want to keep things simple), so we didn't want to introduce any new concepts or tables. Instead, we elected to cheat a little, so please have a look at the example below with a picture being worth a thousand words:
\n![](\"https://lh3.googleusercontent.com/9ofjHjm_CHNEKWH-7C6kdE5aq9ELbeH4ekpOoOZcDMJtwYjP3xWOTiceE3kW1i0pl4H3uTCbh0c5eY1MMqpWJRXjB0J4L66x0vf6YHhs15fY-s_oA-8qobF_eynejn8xlsns09OlmgIGAy7bZNDDebFodPEHmGrPgp8HnAhoCCUIzEi7fV1OWheP\")
Basically we have elected to model a \"has a\" relationship between two Resources using an intermediary Reading to store ids of the referenced (one or many) Resources. In our example a Service Deployment has a Reading Called \"Service Deployment Dependencies\". The main purpose of this Reading is to hold data which we can use to construct id of \"Software Dependency Version\" Resources that our \"Service Deployment\" has. As you can see, if we wanted to obtain vulnerabilities of all dependencies a specific Service Deployment Resource has, we could easily construct ids of each and every \"Software Dependency Version\" and then query for its \"Dependency Version Vulnerability\" Readings.
\nThere aren't any places where we use this not-so-orthodox approach, but there are some and the example below is the most prominent real use-case. We are closely monitoring the performance impact it has on our service and so far, it has been negligible, or at least negligible enough that we are still able to look at ourselves in the mirror.
\nHaving confessed our most prominent sins, let's wrap things up with a less controversial story. There are several items that were not part of the initial design, which we have decided to add along the way, because we have found them useful.
\nThe storage mode for a Reading was a last-minute addition to the picture, but it serves its purpose well. Initially we thought of Readings solely as means of representing the current state of things, e.g. the single-point-in-time data like:
\ntech stack given service is using
\nAWS account id of the deployment
\nIP address of an instance, etc.
\nThis meant that we were only interested in the most recent value of a specific Reading Type for a Resource, always replacing it whenever a new value was collected. From the more technical standpoint - we could get away with performing upserts on Readings with the key being comprised of resourceid and readingtype. However, after a few initial releases, we've discovered that there are certain pieces of data more linear in their nature, which Application Cockpit users could really appreciate on a daily basis.
\nA prominent example of such an item is Service Events Graph, which as you can probably guess represents a service's life cycle timeline, so a mixed bag of Service Deployments, Instance Startups and Shutdowns, Incidents etc. Here's what it currently looks like:
\nWhat we have also found out rather quickly was that it would be quite awkward to represent this kind of data using our initial point-in-time-based approach to Readings.
\nEnter storage mode - a property of a Reading which tells the underlying infrastructure how to behave when a new Reading for the same resourceid and readingtype is persisted.
\nTo preserve all the already existing flows, we have assumed the default mode to be SNAPSHOT, which upholds the default behavior of upserting new values, keeping all the implemented features intact without any additional effort.
\nAs an extension of the functionality, we have introduced TIMESERIES mode, which preserves all the values that were collected and allows us to construct e.g. neat timelines like what you can see in the screenshot above.
\nAs a bonus, here's how the actual data underlying the graph looks like:
\nWhilst it seems quite natural for a Reading to have a timestamp since it is collected (or, as you will, measured) at a specific point in time, it's not so straightforward in case of a Resource, especially not from the Business Domain side of things, since it mainly represents a certain Entity that simply is. Seems like we're not all about Business, I guess, but we'll get to that in a while.
\nWith the above in mind, the first iterations of the Resource model did not have a good reason to include timestamps, so they didn't. However, as we kept adding new features and collectors, we have started noticing that the system is accreting quite a lot of Resources and that a rather large portion of them is only relevant within a short timeframe. It wasn't a problem for us at that point, but even though we hate doing the house chores as much as we do, why not keep our room tidy and \"let go of the things that don't matter anymore\", as we've all certainly been told at least a few times in our life ?
\nThe part where it most stood out for us were Service Instance resources which by their nature are short lived, since as soon as the instance is dead, we don't care about it anymore (apart from audit reasons perhaps). The point was - how do we know whether we can safely delete the Instance Resource ?
\nAs you have seen above, we have ServiceEvents telling us when instances were terminated, but can we rely on the event to always be generated and are we willing to trust it ? Not really.
\nAlternatively, we do track instance health checks but they are not ideal by themselves, particularly since a freshly minted Instance would have no checks at all in the beginning and then report as unhealthy for some time before it becomes stable, not to mention that in some cases it is all right for an instance to fail some of the inspections for a while and still survive. The list of things to consider goes on and on, which went contrary to our goal which was to avoid creating a too-sophisticated cleanup-logic for fear it would become too brittle to be effective.
\nThe introduction of Resource timestamps allowed us to reach all the objectives in a relatively low-cost and simple manner. Once we started tracking when an Instance was created it was a simple manner scanning Instance Resources on a schedule, discovering those which had no health reports for quite a while and were not created recently to safely trim them away from our storage.
\nThe added bonus is that having a Resource creation timestamp comes in handy during investigation of issues (not that there are too many), since it makes it easier to track the timeline and wrap your head around what happened when and why.
\nAt this point I am going to conclude my musings and I hope that you've found them interesting or perhaps even inspiring at some point. The topic is far from exhausted, especially given that our journey with Application Cockpit is still well under way and we keep on adding new features into the application, so please look forward to the next article in series.
","fields":{"slug":"/application-cockpit-model-outline/","tags":["auto1","Cockpit","DataModel","Architecture","Core"]}}},{"node":{"id":"d980aa96-d832-53e1-a6c2-0f3710beb58a","frontmatter":{"category":"Engineering","title":"Application Cockpit Architecture Overview","date":"2022-02-17","summary":"An overview of AUTO1 Application Cockpit architecture and its most notable features","thumbnail":{"relativePath":"pages/application-cockpit/architecture-intro/thumb.png","childImageSharp":{"resolutions":{"base64":"data:image/png;base64,iVBORw0KGgoAAAANSUhEUgAAABQAAAAUCAYAAACNiR0NAAAACXBIWXMAAAsTAAALEwEAmpwYAAADfUlEQVQ4y61UW2tcVRQ+OyAiiIIRVBRT6UPibmIcze2cOff7LZOZzORq0lorMiQxzjgdxQfFB6VFyJPSWKrRNE2wIW3BovlrHrrZy7X3MJGCjxn4OGuvy7e+tfY5oyjn/dN27hNj55ioP9wnMzePycjX98jQF/vkQvcuudA5IIMbe2Rg/WfywrU98lrzgLzePCSX2ifk3c5DMvrZAzK8cUJeXjsiz1TvEpLvk3MXqOjf741o3/1G3//mDvVvHNDxr+7Q4c5tOtb9hY53f6VvNG9RpXKDksYOHVz/ib5yZZeOb+/TydY9Otrcp8Mf/06fr2JO+CN9NtulinXr8ZPo9inzd0+fvHPzIbv47TF79csjNvj5IXupdche/BSxecTe7Dxgb3UesaHWI3ax/Se71P2LjV0/ZW+3/mZDzcfsuaUTpuR/MGXqw21Qr7Zgcm0TSqub+NyG91a2YHRhA7EFY40tKK+1wb5yHdRlzF1ugbrUgpnFFmhLbVAXEQttmJ5vw0i0BUpkWzwLPB6YOne1ae7pM2cIDJX7psotdZK7aEeuwV30W+oEt7UpbmmTEnZZYIp7xjRXoiSB+UYDstkK2K7Xg+Oe2Y7lgWFYkOWzsLL6AURJCkmagReE4MUBxHh2vQAckYtQgiCAWq0GYRiC67oQRREIn7Bdz4VwIwY7tiHAojRN0OdBks/1YnkEOTbyfR8cx5E1im3bkkw4+nYcx9IWPm/RB9M1ZXKC03hYnGaZjAtyz0d1ji3PAopIStMUTNOUDsuyzmwJ3QZd16Xqer0u1YgcERN5Av2zJIzSCsS4kzCKIUxyqeopuA6YlimJxDrkKhCeGB3FiIn665Iju54PPnYXCX7QC4gEUdwfXxALW6jsNxKjr1++Cg280KcIsYDHUcRRLsddcCTm1WqVY3eJLMs4JnLXcbjvudw0DG6aBrctiyMZxyZc18vcskwJRexPSO93Ft1E1/5OhA8J5OtRqTVQGY4Yp4gMLFvcbG+yIIxxLQEoWPAPKijwWSBBgR0LVCVt4ceCAlUVUV4vVj/pFrXlj4q5xctFbeVakVbqRZxXi3SuUVQW1opkdr5QNE0bx1sqCSBJqVwulyYmJqSNNyj9hmGgXyvp2rT06WW1VFaFbci4eBqGLmPn//eFrAOoZgA7/S9EDBVK4Kc1gJ8lwukBYz38dz53gf8CwA4PmL99534AAAAASUVORK5CYII=","width":325,"height":325,"src":"/static/b08a403a7e9534ad64f67f1ad61141df/b3029/thumb.png","srcSet":"/static/b08a403a7e9534ad64f67f1ad61141df/b3029/thumb.png 1x,\n/static/b08a403a7e9534ad64f67f1ad61141df/8d141/thumb.png 1.5x,\n/static/b08a403a7e9534ad64f67f1ad61141df/ee72c/thumb.png 2x,\n/static/b08a403a7e9534ad64f67f1ad61141df/5dfa8/thumb.png 3x"}}},"authorName":"Przemyslaw Walat","authorDescription":"Przemyslaw is an Expert Software Engineer based in our Szczecin office","authorAvatar":null,"headerImage":{"relativePath":"pages/application-cockpit/architecture-intro/1.png","childImageSharp":{"resolutions":{"base64":"data:image/png;base64,iVBORw0KGgoAAAANSUhEUgAAABQAAAAMCAYAAABiDJ37AAAACXBIWXMAAAsTAAALEwEAmpwYAAACaUlEQVQoz0VSS08TURSelQtXrkhI1JjYhQuIIZEC7bzuvTN3bjtDH5TSQlqUBGGlKOGZCOnWP0Di2oAJCTHEgH9O5/Oca6csvsy55/Gdc74zTnjx60/y7Teii/t87utPlEbXmD6+wtTBJab2r/DkwyUe737H9N41nn+6wbOPNyh9vsXM4R1mD+7xau8OT9/f4lH3Gk76A86b3ROUd0/z8s4pZreO8HrrBDObxyitH+Bl/xCl3hHmt85Q3R6h/PYLyptnmB+eY35wjsXhCAsD8m+MMLd2jhe1EziJknlmEqjAy4VbQRR4UH4VEUGHHmLhQrhLiMnWMoDwKggqCwiriwiqCxMIdxHSX4JjanV0VrvIlpcRCkkQCEIxsUNXwnV9JKaG7lrPfqM4hlQRpI4Q6wRCqnG+hBNTsN1uQ2sNKSWSJAH72JZKQu8kCHSIOIpRr9fJr5BmTahIQTcSNBpNRFEEQWRc4wiagkn4wRMxsTEGgqbkWDSM4SvfEmZZRsX0bbQQhiE1VDSttnkFnGRM4PsegsAnBGT7tsCiGsKtuig2iYikiHEeg2sKn2No/HqakRYaupaNO8mHrjS5T42YsFarQREh+/lbpzdvNJGIV5a0ApOx0JE25BQ2gRMZLAd3LgqZjAsVHaW/McDKSmcimSU0JsnHRbmgQkUCt1otewBGmqZ2GiZVdCQrC60pRIher2/JXNe1cV7d4SJaJS9W5Ek6nc5EE/Z5nke/i0G63LRy8CEUoYjz5flA/1eW8i8bFMg5gdfiaz4kk4aeiyRrY317H43uEI3VAVq9dzBpE9qkMFkLWbtv3/8ADCO9cUtyqcEAAAAASUVORK5CYII=","width":1280,"height":786,"src":"/static/477fbbb52f4a6b56a8627202336a0dce/26421/1.png","srcSet":"/static/477fbbb52f4a6b56a8627202336a0dce/26421/1.png 1x"}}}},"html":"Hello fellow concerned developers, or, if you are not yet concerned, while it is not the crux of today’s entry, I hope\nthat by the end of it, you might get interested in becoming one and/or find out how you could do that on your own and\nwhat it actually means for us. Still, the focal point of this post is to fill you in on the architecture of Application\nCockpit (or “App Cockpit”, as we call it in private), the newest tool AUTO1 has brought in for its concerned developers.
\nThe “why”
\nHowever, before dwelling on the “how” too much, let us pay homage to propriety and start with the ‘why’ of the whole\nmatter. The main reason for the emergence of the Cockpit stems from the plethora of tools AUTO1 has adopted to help us\ntrace, measure and monitor our services, not to mention alerting us when things go south. This opulence often means that\nif you are looking for a specific bit of information, whatever the reason, chances are the data is there somewhere, but\nyou need to make a significant effort finding out where to look for it first.
\nTo achieve all that, the Cockpit needs to pull data from multiple sources, aggregate it and present it in a comfortable\nand meaningful manner to the users, not to mention being fast and efficient about it, which leads us to the heart of the\nmatter - the architecture of App Cockpit.
\nFinally, the “how”….
\nWhen designing App Cockpit, we were striving for what probably every software engineer worth his hide can recite even\nwhen woken up at 4 am in the morning - we wanted the architecture to be “open and easily extensible, yet simple and\nrobust at the same time\", preferably allowing polyglot extensions to let every team in the company contribute and share\nwith others if they came up with a cool feature. In order to achieve this, we have split the Cockpit into 5 logical\ncomponents distributed in 3 different layers, as depicted below, with the descriptions to follow.
\nData Collector - the component responsible for fetching data from multiple external sources, parsing, aggregating,\ntransforming and sending it on its way to the data sink with a tiny stretch it might be called the ETL of App Cockpit
\nData Sink - a simplistic component with a one-track mind, whose only job is to receive data in a well-defined\nformat from the Collector, occasionally perform some simple and generic operations on it (like basic validation and\nparent id resolution) and persisting it in the data storage
\nData Storage - a simple storage that holds the current snapshot of App Cockpit data
\nData Provider (aka Backend For Frontend) - exposes an interface to serve everything our UI needs, both serving the\ndata “as-is” from the storage as well as applying some additional logic to further aggregate and transform it for\nspecial purposes
\nUI - yer plain old Front-end - by design contains no logic, it only displays the data as provided
\nSweet, but how about a model?
\nThe specifics of the model run outside of our scope, however, getting it wrong could severely hobble the architecture,\nso it is a story in and of itself, which will have to wait its turn to be told in another blog entry. For our purposes,\nwe will only briefly touch upon it, mainly to establish the domain.
\nTruth be told, App Cockpit is pretty much a simpleton, since its world consists of only two kinds of things: Readings\nand Resources and everything it does revolves around the pair.
\nData Collector - it is the only component for which the logical diagram differs from the physical one, albeit by\ndesign. This stems from the fact that the Collector has been meant as the main extension point that other teams,\npossibly using disparate stacks, can easily plug into when adding new features. By all means, a Collector, as previously\nstated, is any component that fetches data from an arbitrary source, transforms it into Resources and/or Readings and\npushes it to the Data Sink SNS topic. In consequence, when you want to start collecting a new type of data or start collecting it from a different source, you\nhave two options - either add your implementation in one of already existing collector services, or write your own new\none, and the only requirement we superimpose on you is that when you publish to the Data Sink, you follow the format we\nhave specified. At the time this post is being written, we have 3 distinct Data Collector implementations: Main Collector - a general-purpose collector, written in Java and providing data for cross-platform features,\ne.g.: service ownership incidents Docker images used and their vulnerabilities service health and deployments currently running instances code insights (quality, PRs, contributors etc) Java Collector - yet another service written in Java, as the name suggests it collects data strictly related to\nservices written in Java, e.g.: runtime information like JVM and GC being used selective list of JVM parameters passed to the instances egress dependencies for Java services (in this context, the upstream services) PHP/Golang collector - the first contribution coming from outside the original App Cockpit team, written in\nGolang, it is going to provision data for PHP and Golang services, e.g.: egress dependencies vulnerabilities possibly many more to come With a picture being worth a thousand words, let’s have a look at a diagram that gathers all of this in one place: Obviously, the actual process of data collection, as it is currently implemented, is a whole story in and of itself,\ndeserving yet another dedicated blog post, especially since we have elected to make all the collectors stateless, which\nhad its numerous consequences and challenges so keep your eye out for the next entry which is bound to come shortly. Closing words and … we will be back Having said, or rather written all the above, we sincerely hope that we have managed to pique your interest. Not only in\nour solution but also in actually getting “concerned” about your ecosystem, because it does not necessarily have to be\nboring, especially when you have the right tools to make your life easier. On top of that, you can have your fun\nbuilding the actual tools, deftly combining business with pleasure for a change of the usual pace. As hinted numerous times, this is only the preliminary article, meant to lay the groundwork for future additions\nexploring individual pieces in more depth… We are already looking forward to writing those, so keep your eye out!\n
\n\n
\n\n
\n\n
\n\n
\n
![](\"https://lh3.googleusercontent.com/FddBhrlKYHmv4Jw9mucH_Mzo0OUs7kbDEN2xOLqfPm7NCyNcM0fxD0MFSSQHODLDWYL8x-CrwCAj-nTJdeqbZGNk-5MrRwxrg7FI4ILuTQTQe8uEPG7OqHLrRZHnuhnjC9Os3x7KcBEDCaQWGJKk_amQo8GnfBh4PK87vfL6cgGD8buX61uiYEIclQ\")
![](\"https://lh6.googleusercontent.com/mEaBTCaWa1lWEc0cjQraEWVOPMb3V8StdxwrMvaALgxSxMBfjNjk4KBCRif-F_2oUVzgCBheGOpyLiAK5tnxPcH1m2Nae-8EMyi3CgOhoMUSSVAUJY8jFyckMLAq1PMhPI6UPE_uHSEeCy4yNrpX-mPx4he1VywC5lCtkwJBaC77Px36aqiMivkwig\")
![](\"https://lh4.googleusercontent.com/x_T21RH2LmaVQypUzhNgve4ZEPeKBTvWs0IXdJdPF_RVz3EXAb3SK09ZR1JRxmt6OUSTQdRpowIdPGUI0A17-ATiLE-nSY3u8hA0OjZZMO9HF-RCBw9q0I5GG_xIEwMyMy2g5OpxWc7QTmN2KnZy29jJ8pTdt0Av3MNQ4FuKS0e7FT3aRuYZv1diWA\")
![](\"https://lh3.googleusercontent.com/HsXSzSFwhynsa77tvxtyWg2K54pJpTXvXMghLmselpYCFLk7vktBkwWlQ5q7T_WpTiEYSTi85ULtxxz5fNhfleOVEhXf5NRm_51YPAh9x_t1aI53KyHs8QA1bRqLuhjMtZeSXwQIRrOq8rI2FxIxFs2bktIuLclo1ZgQN4sGEcdEKrrW-ZzE10tRHQ\")