Since day one at AUTO1, we strongly believe that the microservice architecture helps us run business at scale not only due to the technical benefits it brings but also, if not foremost by allowing many teams to work independently from each other. Today, we’re running more than 250 microservices in production and on average add one more to our platform every week. All of them are powered by the Spring Framework technology stack, which we value very much for its productivity, but at the same time consider it to be a bit heavy in terms of bootstrapping time. The typical startup time of our Spring applications varies between 40 and 80 seconds. Since we do around 30 deployments per day, our engineering teams spend roughly 30 minutes daily waiting for the deployments to be completed. As we value the time of our engineering teams, we wanted to find a way to decrease these numbers.
\nWhile investigating the startup times of our applications, we have identified classpath component scanning as a potential place to look for improvements. Luckily for us, some of our tech savvy engineers suggested a not so prominent Spring 5.X feature to come at help here - the build time component candidate index.\nThe build time component index is an alternative for regular component scanning, which skips the classpath search operation in favor of using a compile time pre-generated index of component candidates during the spring application context building. However, it comes with a small print from the Spring Framework experts saying that this functionality should have mostly a visible impact on startup times for applications with large amount of beans and that are operating in environments where IO operations are expensive (e.g. remote file systems) or where JVM security managers are in use. Although, neither of this is our use case, we have decided to proceed with the feature evaluation and measurements, as here at AUTO1, we like our decisions to be data driven.
\nThe key part of the indexer feature is a pre-generated index file located in META-INF/spring.components one per JAR. Every index entry in this file is a fully qualified name of a candidate component as a key and comma separated stereotypes as value. So for example “X=Y, Z” can be read simply as register a candidate component X with following stereotypes Y, Z. Below is an example of a spring.components file:
\ncom.auto1.playground.service.SpringDummyService=org.springframework.stereotype.Component\ncom.auto1.playground.domain.SpringDummy=javax.persistence.Entity,javax.persistence.Table,javax.persistence.EntityListeners\ncom.auto1.playground.repository.SpringDummyRepository=org.springframework.stereotype.Component,org.springframework.data.repository.Repository\ncom.auto1.playground.configuration.LocalConfiguration=org.springframework.stereotype.Component\ncom.auto1.playground.service.IndexedCustomService=com.auto1.playground.service.IndexedCustomService\ncom.auto1.playground.Application=org.springframework.stereotype.Component\ncom.auto1.playground.SpringDummyController=org.springframework.stereotype.Component\ncom.auto1.playground.service.IndexedCustomExample=com.auto1.playground.annotation.IndexedCustomThe first question that arises here is how to create such a file and keep it up-to-date. The Spring engineers thought about this problem as well and came up with an annotation processor tool called spring indexer that hooks in during the project build phase and generates the file. Making use of it is fairly simple, assuming you are using your favorite build automation tool, in our case maven and want our IDE to keep this file up to date, simply add the spring indexer annotation processor to your pom file:
\n<build>\n <plugins>\n <plugin>\n <groupId>org.apache.maven.plugins</groupId>\n <artifactId>maven-compiler-plugin</artifactId>\n <version>3.5</version>\n <configuration>\n <annotationProcessorPaths>\n <path>\n <groupId>org.springframework</groupId>\n <artifactId>spring-context-indexer</artifactId>\n </path>\n </annotationProcessorPaths>\n </configuration>\n </plugin>\n </plugins>\n</build>Once this is done, what happens during project build time is the spring context indexer (precisely speaking the CandidateComponentsIndexer class) while going through the project identifies annotated components and outputs them to META-INF/spring.components. The type of components it includes in the candidate list are:
\nClasses annotated directly or indirectly with spring @Component annotation (which by itself is meta annotated with @Indexed) and other stereotypes for which @Component is a meta annotation such as @Repository, @Controller, @Service, @Configuration.
\nClasses and interfaces annotated with annotations from javax package, including CDI annotations (@Named, @ManagedBean), JPA annotations (@Entity, @EntityListeners) or even Servlet annotations (@WebListener)
\nAny custom classes and interfaces annotated with @Indexed annotation.
\nIt should be noted that applying @Indexed on a class/interface where no other standard stereotype (as from points 1 and 2) is present will not make by itself the bean managed, thus not making them eligible for being autowired, but merely provide a hint that it represents a stereotype to be considered in the index.
\nExample of the @Indexed annotation usage:
\n@Indexed\npublic class IndexedCustomService { }Or as meta annotated as below:
\n@Target(ElementType.TYPE)\n@Retention(RetentionPolicy.RUNTIME)\n@Documented\n@Indexed\npublic @interface IndexedCustom { }\n\n@IndexedCustom\npublic class IndexedCustomExample { }In order to make them available in the application context, thus autowirable, we could for example include them in the component scan filters as shown below:
\n@SpringBootApplication\n@ComponentScan(basePackages= {\"com.auto1\"}, includeFilters = {\n @ComponentScan.Filter(type= ANNOTATION, value = {IndexedCustomExample.class}),\n @ComponentScan.Filter(type= ASSIGNABLE_TYPE, value = {IndexedCustomService.class})\n }\n)\npublic class Application implements WebMvcConfigurer {\n // ...\n}There is one big pitfall that comes with the feature though, that is by default the index is automatically enabled when a META-INF/spring.components file is found on the classpath during application start-up and this in turn, disables completely the regular classpath component scanning. In other words, both the index and classpath scanning cannot be used at the same time. The consequence of this is that if an index is available for some JARs, but is incomplete due to including other JARs that come with potential bean candidates but don’t have the index file generated, those beans will not be discovered, thus application startup might spectacularly fail with a well known error: “X required a bean of type 'Y’ that could not be found”.\nFortunately, Spring engineers, as thoughtful as they are, provided a feature flag spring.index.ignore that can be passed either as a system property or set in the spring.properties file at the root of the classpath allowing us to disable the usage of the candidate component index altogether.
\nLike mentioned before, the aim of our investigation was to understand if the Spring indexer feature is of any help to us in regards to boosting the application start-up times. In order to make the test fairly reliable and control as many confounding factors as possible, we have decided to run the test firstly, using a “dummy” Spring 5 application (in two flavors - with 120 and 5000 beans) using all the features a production microservice would use, on a bare metal machine i.e. a MacBook Pro late 2017 i7. Secondly, running a real life production service in a less controlled but close to production environment i.e. on a r4.8xlarge EC2 instance with other services running concurrently. The applications themselves were using Spring 5.0.10.RELEASE with Spring Cloud Finchley.SR2 release train.\nAfter making a couple of hundred measurements of application bootstrapping times with candidate component index enabled / disabled and verifying in the logs that Spring was indeed using (or not) the component candidate list, we ended up with following averaged results:
\nAs the above results have consistently shown, there was no improvement or even insignificant worsening (which will be a subject of our analysis in a separate article) of the bootstrapping time of the Spring applications, regardless of the number of candidate beans present, when using the pre-generated candidate component index. This was expected, as explained in the opening section, the conditions for which this feature was designed (slow or expensive IO operations) are simply not met in our case. One more important factor to consider here, is that a great deal of the beans in our Spring applications, is created using bean definition factory methods, which are not subject to the component classpath scanning, thus are not impacted by this feature.
\nAll in all, we deemed this feature not to be useful for our purposes and moved on to the next item on our list.
","fields":{"slug":"/spring-5-indexer/","tags":["spring","spring-indexer","bootstrapping","performance"]}},"categoryArticles":{"edges":[{"node":{"id":"f3a0fb4b-eb2f-5624-a408-d63e3cee5184","frontmatter":{"category":"Coding","title":"Spring batch integration","date":"2020-02-07","summary":"Usage cases for spring-batch.","thumbnail":null,"authorName":"Artur Yolchyan","authorDescription":"Artur is a Senior Software Engineer at AUTO1 Group.","authorAvatar":{"relativePath":"pages/spring-batch-integration/avatar.png","childImageSharp":{"resolutions":{"base64":"data:image/png;base64,iVBORw0KGgoAAAANSUhEUgAAABQAAAAUCAIAAAAC64paAAAACXBIWXMAAAsSAAALEgHS3X78AAAEfElEQVQ4yz2O2VMbBQDGlzJQSylXIEc3F0eODYGEJAQI5GBzbK49kt0cm703CSRQqfQCB9taap1qgY61VXqoPZxxbKe2dWo7tY6d0RdHX/0HfPLRF991qTPO/OZ7+33fB+RDFjruKMWhcgwqwnYqaiVhKwXbiolRgZiuVxJ1LrsgEg25sFilFqrFBblQFXNVgajyBMCkxiuIs4yMMoiDSTiwoC3kc/hdVpdVPzFqRua9MpupS7mqmK9LVF0u1CVFzssCIXMYwKZcTGpMyWLUOe2yjoxAQyNO1YCutbV9f1tbe1u7dcRY47GaMiXuVVSFvMTjModLLApwaRefHqeibqNm4EBnb/+A1mo2T9kHcZ9tGR5LQocPvdFmNusEOqv8lPicxBESi4kstifzabeMuj1Wnba7i/Q71vDgJ/Xc4zXhyWrxq1pyG/eWvEMd+9umfWMLAiFWshKD/g8goROV5LjdBJa9ls9r2edr4svTjRenG8836o9WK7e46EXM5zbpNGoVX0qKdEqk00qFgkBnAGrOko+Nz1jA94iZhyvFZ8fZ707JLzaWft48/vJ08/6b5Y+K4azHph5QlbCIXE7yJYRXco8U4Ffvn7ZoAiPgRSp8vU59s9Z89s7RJ2fXtxerbxGpo5nIOWxaCLr1BhOZDUqlOF+ICyVEQSwnAcuBFmtHi8ekRT0OMjBZQyI3lvgduVCLzvGx0HJi9nw+WPTbJ4dNdSLMUTBXiAnF+GsSgKN7n62zxWnQVAIuOjCxEJnaxuevleMfkMjJzPxGcuZTPs4FoA0U3uHIIhHkqKgCr1CIAx6wBeptTY3ZTiRnzmPhbTp9gyFuc+hNBnm6Kj48tXy1Ri7Brs8WxTMVvITN8fkIT8I8GRXIKFCAR11g1wSo3WSIb9dXbojEnUb1C5m5XkJ+ePvI4/WVHSm3BE98SKNCZpbBg6xyntzz92QxN+U39XYBQDMdfbTWvCpiF/Kpyyy1Wy/fapS+XC5tcclq0JGdMOViHhYP/Sdzyn5+HuBR3+xwXwcABG1DNxvkOgqfLxR3q4vvoujFQvzeSuGymKV8w5C+m4p7GTTIYEE2F+b2iABcZnLW2tsNAPrOg5f4zJVyYgfH7vHibY76epm8eyR/BpuU0inPNBFwW6RcSPGVfQYPcbl5gM14Ex69/gDQvW8f7rHuyujtGnm3yT84Id45Sp8lw0dwZHPrqX0s3N/Xnwl56UyAI0KvCQN02lOIOcfV7erWFvBgOwwZj+VSm0LleDlfR+aOsfSDV3+yS1udHV2aAZNaBQ4aByddTjQ6VcGCAJv1wX472K9S96gOHeyZmkHoWJiJ+Jsxz9ax1Ve//33p3k86tVmnM2s0JgWtdlCrHdKDw5AVAoio+7DW2NOj6+lRazQj71/99crayY8b+P3rd3/5459Hv/017o7otENGo82gt5jNo0rq9cMGg0UPWoBZj7O3Dxzo1/f1alyT6OL645Xm2cvXflw+93Rz93tx6YJaZRocHDMYIKMRAg9bQHDYZIIMBrvZ7PwXcPlkRtKKCQsAAAAASUVORK5CYII=","width":50,"height":50,"src":"/static/dc9516b9327e627890a1f5c7f84c004a/45876/avatar.png","srcSet":"/static/dc9516b9327e627890a1f5c7f84c004a/45876/avatar.png 1x,\n/static/dc9516b9327e627890a1f5c7f84c004a/eb85b/avatar.png 1.5x,\n/static/dc9516b9327e627890a1f5c7f84c004a/4f71c/avatar.png 2x,\n/static/dc9516b9327e627890a1f5c7f84c004a/9ec3e/avatar.png 3x"}}},"headerImage":null},"html":"Intro
\nIn this article, we will take a look at spring-batch and how it could be used. We will walk through various configurations and we will create an application which reads from a CSV file & writes into a database with outstanding performance.
\nI used the following: Java 11, Spring 5+, Spring Boot 2 and Maven-based project.
\nCreate a Project
\nFirst, we need to create a Spring Boot 2 project. We recommend that you do so by visiting spring initializer, which is a useful tool to generate spring projects with required dependencies and configurations.
\nDependencies
\nWe need the dependencies below to run and test the project:
\n<dependency> \n <groupId>org.springframework.boot</groupId>\n <artifactId>spring-boot-starter</artifactId>\n</dependency>\n \n<dependency>\n <groupId>org.springframework.boot</groupId>\n <artifactId>spring-boot-starter-test</artifactId>\n <scope>test</scope>\n <exclusions>\n <exclusion>\n <groupId>org.junit.vintage</groupId>\n <artifactId>junit-vintage-engine</artifactId>\n </exclusion>\n </exclusions>\n</dependency>\n\n<dependency>\n <groupId>org.springframework.boot</groupId>\n <artifactId>spring-boot-starter-batch</artifactId>\n</dependency>\n\n<dependency>\n <groupId>org.springframework.boot</groupId>\n <artifactId>spring-boot-starter-data-jpa</artifactId>\n</dependency> \n\n<dependency>\n <groupId>org.projectlombok</groupId>\n <artifactId>lombok</artifactId>\n <version>1.18.10</version>\n <scope>provided</scope>\n</dependency>\n\n<dependency>\n <groupId>com.h2database</groupId>\n <artifactId>h2</artifactId>\n</dependency>\n \n<dependency>\n <groupId>org.springframework.boot</groupId>\n <artifactId>spring-boot-starter-test</artifactId>\n <scope>test</scope>\n <exclusions>\n <exclusion>\n <groupId>org.junit.vintage</groupId>\n <artifactId>junit-vintage-engine</artifactId>\n </exclusion>\n </exclusions>\n</dependency>\n\n<dependency>\n <groupId>org.springframework.batch</groupId>\n <artifactId>spring-batch-test</artifactId>\n <scope>test</scope>\n</dependency>\n\n<dependency>\n <groupId>org.hamcrest</groupId>\n <artifactId>hamcrest-all</artifactId>\n <version>1.3</version>\n <scope>test</scope>\n</dependency>spring-boot-starter-batch dependency includes all the configurations to run the spring batch application. Lombok is just a helper dependency to write the code faster and cleaner. H2 is used as an in-memory database. spring-boot-starter-test and spring-batch-test are included for test purposes.
\nBook Class
\nLet’s create a model class book, which will represent a book. This will just serve as a model class and will help us during implementation of spring batch.
\n@Data\n\npublic class Book {\nprivate String title;\nprivate String description;\nprivate String author;\n}Configuration
\nLet’s create a class called SpringBatchConfiguration. We will add all required configurations here. First, let’s annotate this class with @Configuration to be able to inject beans, and with @EnableBatchProcessing to enable spring batch processing. Additionally, we can add @RequiredArgsConstructor from Lombok which would help us to generate constructor with parameters. These parameters are the ones which are marked as final class properties. These properties will be injected as spring beans. Our class will be like this:
\n@Configuration\n@EnableBatchProcessing\n@RequiredArgsConstructor\npublic class SpringBatchConfigurationNow, in SpringBatchConfiguration class let’s add properties which need to be injected into the constructor:
\nprivate final JobBuilderFactory jobBuilderFactory;\nprivate final StepBuilderFactory stepBuilderFactory;jobBuilderFactory and stepBuilderFactory are declared in spring batch jar as spring beans so that we can inject them in any class we want.
\nFile Reader
\nNow, we need to declare and initialize spring beans to configure the batch process. The first bean which we will need will be responsible for reading from a file line by line. Spring Batch provides a default class for it. The class name is FlatFileItemReader. Similarly, spring has different default reader classes for reading from a relational database, mongodb and etc. However, if you need, you can create your own reader and implement it in a way you want.
\nLet’s now see what FlatFileItemReader injection will look like.
\n@Bean\n@StepScope\npublic FlatFileItemReader<Book> bookReader(@Value(\"#{jobParameters['filePath']}\") final String filePath\n) {\nreturn new FlatFileItemReaderBuilder<Book>()\n .name(\"personItemReader\")\n .resource(new ClassPathResource(filePath))\n .delimited()\n .names(new String[]{\"title\", \"description\", \"author\"})\n .fieldSetMapper(new BeanWrapperFieldSetMapper<Book>() {{\n setTargetType(Book.class);\n }})\n .build();\n}We are configuring that the bean reader should read data from the given file path, which should be a CSV file having rows with title, description and author respectively.
\nThe StepScope means to initialize this bean after each step, so file path could be dynamically set when the spring batch job is launched. We will come to that later, how to pass the file path later in this article.
\nItem Writer
\nNow, let’s create a writer class, which will take the data and write into the relational database.
\n@Bean\npublic JdbcBatchItemWriter<Book> writer(final DataSource dataSource) {\n return new JdbcBatchItemWriterBuilder<Book>()\n .itemPreparedStatementSetter((book, preparedStatement) -> {\n preparedStatement.setString(1, book.getTitle());\n preparedStatement.setString(2, book.getDescription());\n preparedStatement.setString(3, book.getAuthor());\n })\n .sql(\"INSERT INTO books (title, description, author) VALUES (title, description, author_surname)\")\n .dataSource(dataSource)\n .build();\n} In the writer bean, we are setting item processors and adding values inside preparedStatement accordingly, which book property should be inserted for each db table column.
\nStep Configuration
\nNow, let’s configure a step which will be executed in the batch process. In our case, the configuration will look like this:
\n@Bean\npublic Step step1(final ItemWriter<Book> writer, final ItemReader<Book> reader) {\n return stepBuilderFactory.get(\"step1\")\n .<Book, Book> chunk(100)\n .reader(reader)\n .processor((ItemProcessor<Book, Book>) book -> book)\n .writer(writer)\n .build();\n}In our scenario, we have only one step, but it's also possible to configure multiple steps. Here, we are creating a spring batch step with the name step1 and setting reader and writer accordingly. These are the readers and writers which we created as spring beans earlier.
\nWe are setting chunk as 100, which means the items chunk proceeded is 100. We can make this configurable too. The processor is for converting the current object to the one which the writer should proceed with. In our case, it is the same object.
\nJob Configuration
\nLast but not least, let’s configure the job which should be executed.
\n@Bean\npublic Job importUserJob(final Step step1) {\n return jobBuilderFactory.get(\"bookReaderJob\")\n .incrementer(new RunIdIncrementer())\n .flow(step1)\n .end()\n .build();\n}Here, we create a job with the name bookReaderJob. We add an incrementer RunIdIncrementer, which is an id generator for the tables which are specifically designed to store the details about job executions in the database. After each time the spring batch job is executed, it will save details about the execution. To check the schema structure for storing this data, take a look at this sql. It is for MySql, but other SQL scripts are available too.
\nAdditionally, we add flow by which the job should be executed. Currently, we have only one step in our flow, so we add it.
\nPlease also add this config: spring.batch.job.enabled=false in your properties config file so that spring batch job won’t be executed automatically with no parameters when the application is started.
Execution
\nTo execute the job, we need to declare JobLauncher and launch the job. To do so, we need to create the below class:
\n@Component\n@RequiredArgsConstructor\npublic class SpringBatchExecutor { \n private final JobLauncher jobLauncher;\n private final Job job; \n\n @SneakyThrows\n public void execute(final String filePath) {\n JobParameters parameters = new JobParametersBuilder()\n .addString(\"filePath\", filePath)\n .toJobParameters(); \n jobLauncher.run(job, parameters);\n } \n}Now, we can call the execute method from whenever we want, with the file path from which the data should be read.
\nAdditional Classes
\nThere are a few additional classes which you will need to execute your code.
\nCreate a class BookEntity so that spring data JPA will automatically create the book table for you. Then you can create repository.
\n@Data\n@Entity\n@Table(name = \"books\")\npublic class BookEntity {\n @Id\n @GeneratedValue(strategy = GenerationType.IDENTITY)\n private Long id;\n private String title;\n private String description;\n private String authorFullName;\n}Then create the interface BookRepository and extend it from JpaRepository. At the moment, we will need this only for testing.
\n@Repository\npublic interface BookRepository extends JpaRepository<BookEntity, Long> {\n} Testing
\nNo one likes a code which is not tested. So, let’s write a few test cases for our class.
\nHere is a code sample for test cases:
\n@Slf4j\n\n@SpringBootTest\nclass SpringBatchSampleApplicationIntegrationTest {\n\n @Autowired\n private SpringBatchExecutor springBatchExecutor;\n \n @Autowired\n private BookRepository bookRepository;\n\n @Test\n public void testExecution() {\n long initialCount = bookRepository.count();\n assertThat(initialCount, equalTo(0L));\n springBatchExecutor.execute(\"sample-data.csv\");\n long count = bookRepository.count();\n assertThat(count, equalTo(7L));\n } \n\n @Test\n public void testLargeData() {\n long startTime = System.currentTimeMillis();\n long initialCount = bookRepository.count();\n assertThat(initialCount, equalTo(0L));\n springBatchExecutor.execute(\"large-data.csv\");\n long count = bookRepository.count();\n assertThat(count, equalTo(60000L));\n long endTime = System.currentTimeMillis();\n log.info(\"executed in miles: {}\", endTime - startTime);\n }\n}The second test executes in ≈ 3500 ms in a machine with a 2.2 Ghz Intel Core i7 and 16GB ram. It proceeds 60K lines of CSV file, and saves it into a relational database.
\nYou can check out the working sample code in github,
","fields":{"slug":"/spring-batch-integration/","tags":["auto1","engineering","spring","code","java"]}}},{"node":{"id":"a3a21c38-873e-56db-a170-eec0e6136040","frontmatter":{"category":"Coding","title":"Writing IntelliJ plugins","date":"2019-12-10","summary":"How to write a simple IntelliJ plugin","thumbnail":{"relativePath":"pages/intellij-plugin/logo.png","childImageSharp":{"resolutions":{"base64":"data:image/png;base64,iVBORw0KGgoAAAANSUhEUgAAABQAAAAUCAYAAACNiR0NAAAACXBIWXMAAAsSAAALEgHS3X78AAAEyElEQVQ4y3WVe0xTVxzHz723rfZBsVeqnc4Yw0aYFSiFFZlZxnSo4w+gSBWEglTmA59jCt4WCOUxq+J4zGwaMqfJfDBi2GbMxjJfMXMzJks0Y5ngfE0mqAwrFLHtPb+de4CKf+wkv5zfOed3Pr/vOb+Te9FAzEokNV9cLvq/Nmpew8L8DIQOANJUPEdh+wGpBB8jrSldo0hd7n0RLJry0fA4bCg2RxYw5S3zm/ISnsWt1vvi8qZMxD1KLkVvb+lmJifS7hrmpJ4lCfidAy8WAqZ8dqzPSwFzIQRN+UEy95SMbxK7MhK3umQiVlnuLVQKwxcVW//SS2O94JNFFZyja3NLHyD03JSHIN5OMxHQZ5C4Fgg0CAlFAIkOAMs6gIWbiV9sQ0cgTlMl+tW7ATSV+E64y2+R9kWVDXJR226PZZSAUvPG5mpxvP0+MfjjjfTg9eh0/Nv8TPGXBdmBS3H5cC52tS8253SffNMN0BZfHdVu7AK+tC9gIIon1MdsvkXvkKojd2aT1N02ZgXDWBnmGBbLJWM5IL4oZzhQEEOMTESsHBBiAjNzz8BrVSJEb721b3taOzdWlIQien+kECfB8gHciM8NkCHOyLLC711d+MPSUkzGIM159njE5IVJ1CfVgXnZbeKbpQ9Ey84+iCy7kxUqylB8QeQoUTeatAWumteIMhKs43m42dMDFotFAgDDMHD+/HlIT88YA7IsNmUcwWmbb4hRu+4BX9mfGQI+SSiY/SBxneeflJLHZ5NKQM7IqKofOzth1qxZMK4QOjo6CDCd+izLwNLlLXhh+R2sr3oEM4SHy8bUOYz07H0daFrQNWPw57XJQG8iIQEwxmC1WiE1NRWampqgu7sbzGYzBUqniLV9gee4B3GE6zHonAMpFAh7ZPSx+sqmH4Pa6XB5++vSHcJbixbhpsZGSEtLA6PRCK2trZCTkxNSK5m+oB2/4h6WYDjcOZgcOvLDHbxyZJfuT6ifDhc3zA8qZIqXNr5sTMiPsJ/CevcoEFgwzPnETGHPBB1VeG+TwQDV2rNnC98FBmlEhUKG1So1TJ06FeRyOchkMuA47iWgzv4t5msCQGB+ldNrDCns3b6AQgGOs42LS67PnR0J+z9pCDY3N+ODhw7i9RvW08qOW8jX2k/jaTUY1M6nI0rhaSSF9e2IRlXvXKD+paJrEbVL9varp6hBGx7m53lejNDxYrhKE6TKGAYj8tiJT4Ea+5mA1o1B5Rz2KwXvnJDClszDtNKfWw9bW7KOwrYl9bAltQGKF3sgb/k+WJpzAGIcXwYiHV/j2Y7vxAjHD4FwxznQurygqQdQunw9yrIBLYV5rEdRQ/YJCty74qujhwp/gt253/R/bP/+eJPj8vvr3JdWvld95X6Spxeiq//Gc2uegMEDoHP7Iaxy5ILKOVREjjtN2k+UIlS34hgF12UeVtRln/i02ta2aWN6q37ydy9mfQdvFK61zqu4CwZnbx8v9Ddrd95PnBxDVI455dknaE9A7OSA+lWn2Orsdi7zo05uYu5V513LTKHXMDHW12Gkcvk4pTBEi6qegLptJ2nfU9CJam1tXKWtjRFWtSND1S06b3H+yiRWXg8l1Ff8y5G3x6oELyJK0cwa6TfwmK79B0KrRX19cxThAAAAAElFTkSuQmCC","width":325,"height":325,"src":"/static/b2f976ed5fa752977d8f8f9b2516534c/b3029/logo.png","srcSet":"/static/b2f976ed5fa752977d8f8f9b2516534c/b3029/logo.png 1x,\n/static/b2f976ed5fa752977d8f8f9b2516534c/8d141/logo.png 1.5x,\n/static/b2f976ed5fa752977d8f8f9b2516534c/ee72c/logo.png 2x,\n/static/b2f976ed5fa752977d8f8f9b2516534c/5dfa8/logo.png 3x"}}},"authorName":"Piotr Czekaj","authorDescription":"Piotr is a Senior Software Engineer at AUTO1 Group.","authorAvatar":{"relativePath":"pages/intellij-plugin/avatar.png","childImageSharp":{"resolutions":{"base64":"data:image/png;base64,iVBORw0KGgoAAAANSUhEUgAAABQAAAAUCAYAAACNiR0NAAAACXBIWXMAAAsSAAALEgHS3X78AAACyElEQVQ4y4VUO2gUURR9mZmN8YcItoKVjeXuzsxu9jOzm92NkBBTqDEmjZAEgtqIpSgI1gp+uhS6kEI7wc7GQsFCLERiISGCaGHhF02yu+M5s/cujyC4cJY379573r3n3XdNvV51xsYiJ4rKbqlUyBj8wjAYyWazp4BVYB3YAraBDeAhMBcE/l76lkrFTLVachuNyCGXEbJMrVbx6ADnaWAtn88nvu8nWBMdQcJ9Auv3wCxjWq2aC9IMSF3DzCyyG1YAs9pSYkL2melmLpfT71tKGkUlz2iZJAuCgA5dEjGAwPoV0AbuAy+tfR7YkZjb5CiXR1OutEw5jWQdCfgCnEiSxNg/7LWAj+KTSiHSzBvrAtascrriPCmnDkP4AYQ0Em3TGPHf8P38ARpP/4PsHQ7aU69Hqdi4OJeI47JXLg9IX1tVbUuWCzSsygcJO+L0goGNRpySoB1colarerhEvcCn4tuxCB/TsC5ZdS3Ct2Ho75uZmRoigWbIbgCxt7Q0O8TLsgg17o2R2+JHzzJcV/1IoBlybel42SLsCcc3I6UqoaZ+lUHFYjjCzJSQa+4J4UVLKiX8ScOGVXJXDN+xd7SfZXFYS65U+n2Wz+cOw+ezHadSkfCRdVJiZdlmMG9ZtHMpgWR3Z0eMSvWExjmrbXoCbZ951dIim7Yk6u1IYtlwavChW51Pp01tXOwf01eC9RHs/Rbbnx2Hf4IUh/Q5zdol8H3KG22Pj9edOK54LF1876pd37TELhqdZ+J4MwxDHUvXgDOFQrB7YqLpjI4WdhG8GLYNbCeBK/1+TWNWRG+Pfy5Hj5Dek3IXzX9+8DkL/AIeLC/PDU1NHXdSQk5altNsDkjPAT+ArzKyFnK5bB2IxbYik4hanmfM5GTL6Q/Y2DUc25y0HLToubR8TI39cL4APJOxr/35AXgOXEJnHKQvepP6eiQj119kDrZjqLo8HAAAAABJRU5ErkJggg==","width":50,"height":50,"src":"/static/26a7a327ccc4b335712e5a7086f2b26d/45876/avatar.png","srcSet":"/static/26a7a327ccc4b335712e5a7086f2b26d/45876/avatar.png 1x,\n/static/26a7a327ccc4b335712e5a7086f2b26d/eb85b/avatar.png 1.5x,\n/static/26a7a327ccc4b335712e5a7086f2b26d/4f71c/avatar.png 2x,\n/static/26a7a327ccc4b335712e5a7086f2b26d/9ec3e/avatar.png 3x"}}},"headerImage":{"relativePath":"pages/intellij-plugin/header.png","childImageSharp":{"resolutions":{"base64":"data:image/png;base64,iVBORw0KGgoAAAANSUhEUgAAABQAAAALCAYAAAB/Ca1DAAAACXBIWXMAAAsSAAALEgHS3X78AAADGklEQVQozx3R60/bVRjA8V80m2N4IVkWcfpmqyulTdd7oRYopEDbyaXlVgptobANkAzrUpwbyjZvgMpQ7CxhGCaXUTcv04mRRYMkKpB1MLYpIKkymb5S/4WvZ7745DkvnnOey5FS87qQWbqY0zWyIEzrmxkxHOFHcXZkRZCsZ0ixnmKH9TQPWV7m0eIo22q/JmjvI6ENsijyh7I62G7rYXt2F1KgpIo2TZhlbYBFkTCjDaE2Rdhn6mS36TiPC+nml9ilP0a6rQ91wxwZhxJ4ar9kMbuDdeMR+h0DyAOzyBwxpJDhMCsHAlzXBFgQ7mj8DGqaeUr3Agrt8yh0YWQiZolOq9wTuLxXcFZ/SmHNFVrKL/DKwXMcrPkcu2+aYkGKqkKsquu5rvZzQ7gf19R1nFTUo1M0Y5E3YNWGidu6mSzsQdc0j+ZwgrzgLFUVcWEKn/uiMInfPY40pPKxrvWypPFyU7ilqWZJX8dr9l4UrjFUzlFy3OJS8BqWtmVkRzeQtf/K3qNJCsT4EWeUU65+ely9HHOdQ3pR4Wa1OI8Vez637DaWiwp5tfxtIidmGbywxsilJH2xn3lneI3RyXU+iK3Q1votPu9VfP5vOOGMEbeHmXEcoregV+wwvYx4volk816SoSeZrS3lfPmbfFj2OqPeQcbqo4z7+pnwDvBxYJjJ2iFGSt8j5jhLzPUuY7ndXLREmMruZMJyHCmyq4jO3UWMGg1MO+XM5z7LkiHEDwcamVM28JM6wFpODas5Xm7omljRt3DH1Mptcxu/GFv4yNlNZ3CQPk8P14ytSG+l5XHmESund+ZzcmcuX+xx8ldmNckMD78pKthUevjHXMK/5lLuZlYIlWwpq9lSVJJUeQkFL2N77ntsHfO8URpFGk41M572DLFUI++nGDj/sJnb+0q4l1nFH8LW/yrZ2O8mKfdwVxT5/X4x4c8MN+3ip1XtGxj8X+GouYw081gW9+TlfJaiY/xBJWMPKPlkh5bEEwXc3GMnkV7Ad2nZxLepuZpqYHN/GVvisU3h76dLmDKGsTQuoK+7RJNrgP8AU8rtTZmilCkAAAAASUVORK5CYII=","width":1280,"height":720,"src":"/static/6fae10ea7503a11a8f308ac2e2a5217a/26421/header.png","srcSet":"/static/6fae10ea7503a11a8f308ac2e2a5217a/26421/header.png 1x"}}}},"html":"During analysis of why some services take a long time to build, several problems have been found, mostly related to number of Spring contexts that were created during integration tests. More about those problems can be found in previous blog post. In order to decrease the likelihood that slow tests will appear at AUTO1, we implemented an IntelliJ plugin which warns a developer about possible problems when he writes code and provides quick fixes if they are applicable.
\n@DirtiesContext annotation which slows tests down because it forces creation of new Spring context@ActiveProfiles (slows tests down because different Spring context would have to be created for each combination)@ActiveProfiles is greater than 3@FeignClient (or DTO used by client) not documented with Swagger annotations@TransactionalDuring the plugin development, we realized that resources on writing IntelliJ plugins are scarce, and therefore decided to create a step by step tutorial on how to implement @DirtiesContext inspection.
In our tutorial we will be using the community edition of IntelliJ 2018.3.5. Final source code is available on github.\nThere are several ways on how to create IntelliJ plugin. The recommended one is to use Intellij plugin for Gradle and that’s what we will use in this tutorial. Let’s start with creating new project using File > New > Project..., select Gradle and make sure that in Additional Libraries and Frameworks both Java and Intellij Platform Plugin are selected.\nIf you cannot see “Gradle” or “Intellij Platform Plugin” then please make sure that both “Gradle” and “Plugin DevKit” IntelliJ plugins are installed.\nContinue next steps of the wizard using default values, use any groupId and artifactId. After some time IntelliJ will download dependencies and create an empty plugin.
\nOur inspection should warn a developer each time there is a class annotated with org.springframework.test.annotation.DirtiesContext and provide quick fix which deletes @DirtiesContext. If Spring is not your thing then any other annotation can be used instead.
There are two kinds of inspections:
\nLocal inspections are executed in the background when file is opened, in general they have access to currently open file. Local inspection cannot report problem for not currently processed file. Inspection class has to extend com.intellij.codeInspection.LocalInspectionTool or one of subclasses.\nGlobal inspections work only in batch mode when analysis is manually triggered via Analyze > Inspect Code and see complete graph of references between classes and can report problem for any file. Inspection class has to extend com.intellij.codeInspection.GlobalInspectionTool or one of subclasses.
We don’t need to access complete graph of references and we would like to give a hint that something is wrong as soon as possible so local inspection is a better choice for our use case. First problem that we encounter is to pick proper base class. One approach is to ask IntelliJ to show class Hierarchy of LocalInspectionTool and take a look what other inspections are extending. In this case AbstractBaseJavaLocalInspectionTool seems to be a good choice since most of Java inspections are based on it.
Create new class named DirtiesContextInspection with following content:
public class DirtiesContextFirstVersionInspection extends AbstractBaseJavaLocalInspectionTool {\n private static final String DIRTIES_CONTEXT = \"org.springframework.test.annotation.DirtiesContext\";\n\n private static final String DESCRIPTION_TEMPLATE = \"Usage of @DirtiesContext makes integration tests slower\";\n\n public String getDisplayName() {\n return \"Usage of @DirtiesContext is not recommended\";\n }\n\n public String getGroupDisplayName() {\n return GroupNames.PERFORMANCE_GROUP_NAME;\n }\n\n public String getShortName() {\n return \"DirtiesContext\";\n }\n\n public boolean isEnabledByDefault() {\n return true;\n }\n\n public PsiElementVisitor buildVisitor(@NotNull final ProblemsHolder holder, \n boolean isOnTheFly) {\n return new JavaElementVisitor() {\n\n public void visitAnnotation(PsiAnnotation annotation) {\n super.visitAnnotation(annotation);\n\n String qualifiedName = annotation.getQualifiedName();\n\n if (DIRTIES_CONTEXT.equals(qualifiedName)) {\n holder.registerProblem(annotation, DESCRIPTION_TEMPLATE);\n }\n }\n };\n }\n}The most interesting thing happens in a visitor which is notified each time a Java annotation is encountered in source code. By overriding different methods, plugin can process methods, classes, imports etc. In our case we check if fully qualified name of annotation matches our expectations and register a problem when that’s the case. Later on we will change this class to include also a quick fix. Creation of inspection class is not enough to make it available in IntelliJ - it’s also needed to register inspection in plugin.xml which among other things describes what plugin does, what other plugins are required and in which version of IntelliJ it can be used.
It’s possible to register each inspection one by one in plugin.xml under extensions tag and configure inspection using xml but we find it easier to register inspectionToolProvider and configure using it in Java code.
Create the following class to implement our inspection provider:
\npublic class CodeInspectionProvider implements InspectionToolProvider {\n public Class[] getInspectionClasses() {\n return new Class[]{\n DirtiesContextInspection.class\n };\n }\n}And register it in plugin.xml:
<extensions defaultExtensionNs=\"com.intellij\">\n <inspectionToolProvider implementation=\"com.auto1.intellij.tutorial.CodeInspectionProvider\"/>\n </extensions>Now it’s time to check our inspection in action by running runIde gradle task. This could be done from terminal by executing ./gradlew runIde but it’s better to use Gradle view in IntelliJ since it allows to start IDE in debug mode (if needed just right click on task and select Debug, you can create run configuration to speed-up in the future). New IntelliJ instance should show up, if you already have sources of some project that uses Spring Boot then open it, otherwise you could create a new project. Annotate some class with @DirtiesContext annotation and observe inspection marker showing up. In case of problems logs can be found at build/idea-sandbox/system/log/.
Many inspections report not only problems but also provide automatic ways of fixing issues. In case of DirtiesContext, it’s not possible to provide safe way of removing it because DirtiesContext is often used when bean holds some state which makes tests dependent on each other. Usually we want to remove the annotation and then figure out the \"dirty parts\" and clean them up in an elegant way. Since the second part is hard to automate we will provide quick fix which only removes annotation.
\nGo back to DirtiesContextInspection class and add quick fix:
private static class DeleteQuickFix implements LocalQuickFix {\n public String getName() {\n return \"Removes usage of @DirtiesContext\";\n }\n\n public void applyFix(@NotNull Project project, @NotNull ProblemDescriptor descriptor) {\n descriptor.getPsiElement().delete();\n }\n\n public String getFamilyName() {\n return getName();\n }\n }Next step is to pass quick fix when problem is registered:
\nholder.registerProblem(annotation, DESCRIPTION_TEMPLATE, new DeleteQuickFix());Executing runIde Gradle task should prove that quick fix works as expected.
Up to this point we have used hardcoded strings inside inspection name and description. To allow the plugin to be accessible in different languages we can externalize the messages. For this purpose we can use properties file. First create PluginBundle class:
public class PluginBundle {\n private static Reference<ResourceBundle> ourBundle;\n\n private static final String BUNDLE = \"com.auto1.intellij.tutorial.PluginBundle\";\n\n private PluginBundle() { }\n\n public static String message(@NotNull @PropertyKey(resourceBundle = BUNDLE) String key, \n @NotNull Object... params) {\n return CommonBundle.message(getBundle(), key, params);\n }\n\n private static ResourceBundle getBundle() {\n ResourceBundle bundle = com.intellij.reference.SoftReference.dereference(ourBundle);\n if (bundle == null) {\n bundle = ResourceBundle.getBundle(BUNDLE);\n ourBundle = new SoftReference<>(bundle);\n }\n return bundle;\n }\n}Then create a new file matching BUNDLE constant, in my case it will be src/main/resources/com/auto1/intellij/tutorial/PluginBundle.properties with content:
inspection.dirties.context.display.name=Usage of @DirtiesContext is not recommended\ninspection.dirties.context.problem.descriptor=Usage of @DirtiesContext makes integration tests slower\ninspection.dirties.context.use.quickfix=Removes usage of @DirtiesContextRegister bundle in plugin.xml:
<resource-bundle>com.auto1.intellij.tutorial.PluginBundle</resource-bundle>And finally use bundle in inspection, for example:
\n public String getDisplayName() {\n return PluginBundle.message(\"inspection.dirties.context.display.name\");\n }We picked LightPlatformCodeInsightFixtureTestCase as base for our tests because it is recommended in the documentation. Unfortunately, testing appeared harder to set up properly than expected.
First problem was that our tests couldn’t see classes from JDK, which was fixed by specifying project descriptor to use internal JDK:
\n protected LightProjectDescriptor getProjectDescriptor() {\n return new LightProjectDescriptor() {\n public Sdk getSdk() {\n return JavaAwareProjectJdkTableImpl.getInstanceEx().getInternalJdk();\n }\n };\n }Second problem was that visitor received incorrect fully qualified class name of the annotation. Instead of org.springframework.test.annotation.DirtiesContext, it got DirtiesContext while it worked fine for real project in IDE. It turns out that such behaviour occurs when test project doesn’t see definition of some class. This is fixable by either hardcoding problematic class into test or by adding dependency as library to the project. Second approach avoids copying source code from other projects and seems to be more interesting so it will be presented here. In order to download dependency jar we use ShrinkWrap library:
private File[] getMavenArtifacts(String... mavenArtifacts) {\n File[] files = Maven.resolver()\n .resolve(mavenArtifacts)\n .withoutTransitivity()\n .asFile();\n if (files.length == 0) {\n throw new IllegalArgumentException(\"Failed to resolve artifacts \" + Arrays.toString(mavenArtifacts));\n }\n return files;\n }When dependency is resolved and downloaded into local Maven cache it can be added as library with code listed below:
\n protected void attachMavenLibrary(String mavenArtifact) {\n File[] jars = getMavenArtifacts(mavenArtifact);\n Arrays.stream(jars).forEach(jar -> {\n String name = jar.getName();\n PsiTestUtil.addLibrary(myFixture.getProjectDisposable(), myModule, name, jar.getParent(), name);\n });\n }It’s important to use myFixture.getProjectDisposable() instead of myFixture.getProject() otherwise there is an exception during test shutdown:
com.intellij.openapi.util.TraceableDisposable$DisposalException: Virtual pointer 'jar:///somePath/.m2/repository/org/springframework/spring-test/5.1.5.RELEASE/spring-test-5.1.5.RELEASE.jar!/' hasn't been disposedNext surprise is that by default test searches for test data in strange location inside IntelliJ home folder which can be fixed with overriding getTestDataPath. Since input files most likely won’t compile because of missing imports and possible usage of special markers like src/test/java folder to store them:
@Override\n protected String getTestDataPath() {\n return \"src/test/testData\";\n }Inspection tests can be done by providing file to analyse and resulting file that should be created after given quick fix has been applied. Inspection tests use configureByFile to load input file, doHighlighting to trigger source code analysis, launchAction to execute quick fix and finally checkResultByFile to compare results against after file.
myFixture.configureByFile(testName + \".java\");\n myFixture.enableInspections(new DirtiesContextInspection());\n myFixture.doHighlighting();\n IntentionAction quickFixAction = myFixture.findSingleIntention(intentionHint);\n myFixture.launchAction(quickFixAction);\n myFixture.checkResultByFile(testName + \".after.java\");Complete source code can be found at github github
\nIf you would like to share your plugin with other developers, you can publish to JetBrains plugin repository as described in the documentation. Other simple option is to execute buildPlugin Gradle task which will create plugin zip file inside build/distributions and then install it via Install Plugin from disk... available inside Preferences > Plugins (in IntelliJ 2018.3 is available through \"gears icon\").
\nWhen working on your own ideas you might run into a situation when you don't know how to implement some functionality. In this situation you could try to find the answer using links provided in section below. What also worked for us was reading source code of inspections available as part of community edition of IntelliJ, often there is an existing inspection which does a similar thing to what you might want to do.
\nThe PostgreSQL team announced recently a new release of the most advanced\nopen source relational database - PostgreSQL 12. As usual it comes with an impressive list of improvements\n(generated columns ♥️), one of them being long awaited by dozens of developers: improved Common Table Expressions.
\nI should first explain what are the Common Table Expressions for those who are unfamiliar with them:\nthe CTE’s, often called “WITH queries”, are SQL constructs giving a possibility of creating temporal data views for a sake of a query execution.
Essentially CTE is an additional query which results can be referenced in the subsequent CTE’s or the main query before which\nit is being placed. It should be clear enough with an example - here’s a sample query with two CTE’s taken from Postgres docs:
\nWITH regional_sales AS (\n SELECT region, SUM(amount) AS total_sales\n FROM orders\n GROUP BY region\n), top_regions AS (\n SELECT region\n FROM regional_sales\n WHERE total_sales > (SELECT SUM(total_sales)/10 FROM regional_sales)\n)\nSELECT region,\n product,\n SUM(quantity) AS product_units,\n SUM(amount) AS product_sales\nFROM orders\nWHERE region IN (SELECT region FROM top_regions)\nGROUP BY region, product;Now we know what they are, but what purpose can they serve us? Well - we could parry here and say: for the same purpose as ordinary database views serve.\nThat’s of course a dramatic simplification - CTE’s are much more powerful and should not be treated as a simple database views.\nNevertheless I would like to keep the collation for the sake of this article.
\nDatabase views are absolutely optional - one can simply substitute them with a subquery and achieve identical results.\nIndeed that’s what modern database engines do these days - once a database view is being used they inline it’s query as a subquery.\nWhy to bother then? We are able to deal completely without database views and even if we did use one the database engine would get rid of it anyway.
\nWhat benefits do views give us then? And why the heck are they inlined?
\nThe most appropriate explanation here is that database views help us achieve better readability.\nThey offer an elegant way of abstracting some parts of a database into a meaningful object, often matching closely with the domain.
\nInstead of creating giant and ugly looking queries it is possible to extract some of its parts into an appealing view which is easier to browse and select data from. \"Divide and conquer\" rule in it’s true form.
\nStill, we didn’t answer the fact that the underlying view’s query is most of the times inlined while it is being referenced. The reason is performance of course. Smart guys found out that lazy evaluation helps the optimizer a lot - by delaying the execution we could take advantage of a context of the actual query.
\nThis in turn allows many clever optimization techniques, like: pushing down predicates (WHERE filters), eliminating unnecessary JOINS, accessing only subset of columns etc.\nIn other words - a database is smart enough to do as little work as possible when evaluating database views, in the context of the issued query.
\nPersonally I love this pattern: aggregating all the data into views and letting a database to optimize my queries - these folks are really good in it and my queries are dead simple too.
\nI have mentioned the CTE’s at the beginning, saying they are able to create temporal data views. I still conform to the comparison with database views - they both are in many cases similar.\nThe main difference is that CTE results are temporary and are reachable only in the context of a query which CTE is being part of.
\nIt may make sense to use a CTE in a place where a regular database view is not justified (e.g. it makes sense only in the context of a query and not in the whole domain), expecting similar behavior.
\nBesides many remarkable advantages there’s at least one disadvantage which disqualifies CTE's for most use cases - before PostgreSQL 12 it was implemented as an optimization fence. What does it mean?
\nEasy to imagine an example with a generic data view aggregating lots of data. If the view is then used to select just few records it could mean a tremendous waste of computation, if the aggregation is executed immediately.\nInstead the aggregation should be executed only on a small subset of data, which can be deduced from the outer query.
\nUnfortunately such counter intuitive behavior was true for CTE’s for a long time - their results were materialized only to be accessible for the rest of the query afterwards.
\nNot to blame anybody - the creators had quite good reasons to implement such behavior (i.e. guarantee of exactly one evaluation, possibility of a recursive CTE’s and more) but this still feels like focusing on corner cases instead of optimizing the happy path.\nThat’s why the community insisted for a long time for changing the status quo by giving the possibility of disabling the fence and unlocking the full potential of CTE’s.
\nThe SQL gods listen to their prayers and here it is - PostgreSQL 12 with updated Common Table Expressions.\nBy default, when few constraints are met, the queries will be inlined allowing joint optimizations.
\nIt is still possible to force the old behavior - by defining the CTE AS MATERIALIZED the engine would execute it immediately.\nIt is also possible to hint the optimizer that we definitely want the CTE to be inlined, e.g. when the CTE is being referenced twice the engine won't inline it by default.
This is truly a game changer for many developers who care about their queries’ readability, allowing them to substitute not very well liked subqueries with elegant “WITH queries”.
Don’t get me wrong - not every subquery should be immediately replaced, they still have their strengths and in some cases they should be chosen over CTE’s. It is just convenient to have two distinct tools in a toolbox, isn’t it?
","fields":{"slug":"/postgres12-a-precious-release/","tags":["postgres","postgres12","release","sql","cte"]}}}]},"authorArticles":{"totalCount":2,"edges":[{"node":{"id":"4762bbde-b461-512b-93ae-22335b823a90","frontmatter":{"category":"Architecture","title":"Harmonizing Application Cockpit data using DB views","date":"2023-02-16","summary":"Our journey building the AUTO1 service inventory search using DB views","thumbnail":null,"authorName":"Mariusz Sondecki","authorDescription":"Mariusz is a Expert Software Engineer based in our Szczecin office","authorAvatar":{"relativePath":"pages/harmonizing-application-cockpit-data-using-db-views/avatar.jpg","childImageSharp":{"resolutions":{"base64":"data:image/jpeg;base64,/9j/2wBDABALDA4MChAODQ4SERATGCgaGBYWGDEjJR0oOjM9PDkzODdASFxOQERXRTc4UG1RV19iZ2hnPk1xeXBkeFxlZ2P/2wBDARESEhgVGC8aGi9jQjhCY2NjY2NjY2NjY2NjY2NjY2NjY2NjY2NjY2NjY2NjY2NjY2NjY2NjY2NjY2NjY2NjY2P/wgARCAAWABQDASIAAhEBAxEB/8QAGAABAAMBAAAAAAAAAAAAAAAAAAMEBQH/xAAXAQEBAQEAAAAAAAAAAAAAAAACAwAB/9oADAMBAAIQAxAAAAGv2Svu3GUctOAU5wh//8QAGxAAAQUBAQAAAAAAAAAAAAAAAQACERIiEyH/2gAIAQEAAQUCJkDB7KBW+bEJwkNZ6v/EABgRAQADAQAAAAAAAAAAAAAAAAEAAhEQ/9oACAEDAQE/AQMmMLPP/8QAGBEBAAMBAAAAAAAAAAAAAAAAAQAQESH/2gAIAQIBAT8BV2cmFf/EAB4QAAEEAQUAAAAAAAAAAAAAAAABAhEhQRAxUWFx/9oACAEBAAY/AqwSjpMj+oKvRU5PDZD/xAAdEAACAgIDAQAAAAAAAAAAAAABEQAhQWExcZFR/9oACAEBAAE/IWdbBmVNkJTg+IpKCEBHylkKGgktmegIYKT9dwgOydif/9oADAMBAAIAAwAAABAA18H/xAAXEQEBAQEAAAAAAAAAAAAAAAABEQAQ/9oACAEDAQE/EES8YEurv//EABgRAAIDAAAAAAAAAAAAAAAAAAEQESEx/9oACAECAQE/EDJJlS//xAAdEAEAAgICAwAAAAAAAAAAAAABABEhQVFhMZGx/9oACAEBAAE/ECQu0QVxCvgawa6iFVCFbUJDi/sW7HoRRveCLN+4F4gKruoOH4KtotWHILP/2Q==","width":50,"height":50,"src":"/static/41d7e93878c0ba2eb12594d317c2b361/d2d31/avatar.jpg","srcSet":"/static/41d7e93878c0ba2eb12594d317c2b361/d2d31/avatar.jpg 1x,\n/static/41d7e93878c0ba2eb12594d317c2b361/0b804/avatar.jpg 1.5x,\n/static/41d7e93878c0ba2eb12594d317c2b361/753c3/avatar.jpg 2x,\n/static/41d7e93878c0ba2eb12594d317c2b361/31ca8/avatar.jpg 3x"}}},"headerImage":null},"html":"As you may have read about Application Cockpit in one of our previous articles, it is “The tool which lets concerned developers quickly find their bearings and gather relevant information pertaining to a particular service running in our ecosystem”.
\nOver time, AUTO1 engineers and managers grew fond of using Application Cockpit, and started relying on it on a daily basis. The increase in popularity, naturally, led to development of new features, out of which, one of the arguably most impactful ones was the \"dynamic service search\" (or \"App Cockpit Search\", as we call it internally).
\nAUTO1 platform currently consists of 600+ microservices, and on average two new services are created every week. Many times in the past, we needed to search our AUTO1 service inventory for services which were matching some specific criteria, e.g. are written in a specific language, using certain frameworks or libraries, or have certain features enabled.
\nAs it happens, this is actually where App Cockpit App Cockpit Search chimes in. From the user’s perspective, it is an UI that lets the user input some search criteria using the Cockpit Query Language (or CQL for short) and be presented with a list of services matching the criteria. One typical use case is searching the AUTO1 service inventory for the usage of a specific library in a specific version.
\nBelow is an example search for services using org.springframework.boot:spring-boot-starter dependency in version 2.3.0 onwards:
Once we were certain of what we wanted to achieve, i.e.: search our AUTO1 service inventory using dynamically defined criteria, we began the efforts of defining the architecture.
\nAs you may recall from our previous article about the Application Cockpit, we use Backstage as our Front-end. Backstage is fetching data from our backend-for-frontend service called “Application Cockpit Data Provider”, which in turn, is backed by a PostgreSQL DB.
\nAs Backstage itself comes with search functionality that could be potentially used for our purposes, we were left with two options, either to reuse the already provided solution and tailor it to our needs or create a solution from scratch.
\nAfter evaluating the Backstage search, which at that time, was still in the early alpha stage, it was clear to us that it would be a bumpy ride. As it is a full-blown search mechanism, or to put it more bluntly, an over-blown mechanism, compared to what we really needed:
\nAs we already use Spring Data JPA as the persistence layer to communicate with PostgreSQL in our data provider service, we have limited our options to two choices: dynamically built queries on top of joined database tables or database views. Either of them seemed to us, initially, as a good candidate for dealing with the complexity that stems from the heterogeneous data structures.
\nThe idea behind this solution was to create a set of SQL queries, backed by corresponding Spring Data JPA repositories, one for each attribute in question, execute these queries against the DB, and then combine the results on the Java side performing the necessary filtering and aggregation.
\nThe advantage of this solution was that we would keep things small and isolated. Each query is in a separate repository. Each one would deal just with the data structures it's aware of. On the other hand, this solution would come with a significant performance penalty, due to the substantial amount of data that would be transferred from the DB, just to be later filtered out on the application side.
\nThe alternative solution would be to dynamically build a large SQL query that would perform both the retrieval of the data and the filtering and aggregation of its results.
\nHowever, this approach would come at an increased maintenance cost. As every data structure change would require a corresponding Java code change, hence we would lose the benefits of having the data values stored in the JSON objects. On top of that, maintaining a large SQL query would quickly become a problem on its own.
\nKnowing the shortcomings of the dynamically built queries, we decided to evaluate another promising approach - the database views, that could hide the complexity of dealing with the heterogeneous data structures:
\nIn that approach, we would create a database view that could look like the following (based on our previous example):
\nwhich returns:
\nThe main database view, here called document_view, would be the aggregation of multiple sub-views. Each sub-view would only know about its data structures, hence we would achieve the desired single responsibility and encapsulation traits, making the solution more extendible and more maintainable. Even if something would be changed within one of the sub-views, it would not affect the aggregated view or other sub-views.
As previously mentioned, since we use Spring Data JPA extensively, the database view would be reflected in the code as a JPA entity which would allow us also to use the Criteria API to build a generic query on top of that main view and utilize the built-in PostgreSQL functions, that would come especially handy when interacting with the JSON objects.
\nAs the DB views seemed like a good solution, we still needed to answer one more question: whether to materialize them or not. As materialized views would be, obviously, faster to query, but that would not come for free, as we would need to build a mechanism to frequently refresh the materialized views, which would also induce a performance overhead. In the end, as we expect the App Cockpit Search to happen fairly infrequently and we value accuracy over query execution time, we’ve decided to not materialize the views.
\nFinally, we decided to move forward with the DB view based solution. Although it’s not a silver bullet and requires us to keep an eye on the query performance, as over time the number of sub-views will grow, which might impact the overall search performance, we consider this to be a fair trade-off for the gained extensibility and maintainability of the search functionality.
","fields":{"slug":"/harmonizing-application-cockpit-data-using-db-views/","tags":["Cockpit","PostgreSQL"]}}},{"node":{"id":"89a8474c-7217-53ee-bffa-b7a1d7f79b26","frontmatter":{"category":"Engineering","title":"The OpenAPI journey","date":"2021-11-22","summary":"A story on how OpenAPI helped AUTO1 Tech to move forward faster and in a more structured way.","thumbnail":null,"authorName":"Mariusz Sondecki","authorDescription":"Mariusz is an Expert Software Engineer based in our Szczecin office","authorAvatar":{"relativePath":"pages/openapi-journey/avatar.jpg","childImageSharp":{"resolutions":{"base64":"data:image/jpeg;base64,/9j/2wBDABALDA4MChAODQ4SERATGCgaGBYWGDEjJR0oOjM9PDkzODdASFxOQERXRTc4UG1RV19iZ2hnPk1xeXBkeFxlZ2P/2wBDARESEhgVGC8aGi9jQjhCY2NjY2NjY2NjY2NjY2NjY2NjY2NjY2NjY2NjY2NjY2NjY2NjY2NjY2NjY2NjY2NjY2P/wgARCAAUABQDASIAAhEBAxEB/8QAGAABAQEBAQAAAAAAAAAAAAAAAAQDAQX/xAAXAQADAQAAAAAAAAAAAAAAAAABAgME/9oADAMBAAIQAxAAAAGbumAap5ikb5hWqFs3/8QAHRAAAgEEAwAAAAAAAAAAAAAAAAECAxETIhIyM//aAAgBAQABBQJ7KyiZB+eTXkVOkRpX/8QAGREAAwADAAAAAAAAAAAAAAAAAAERAhAx/9oACAEDAQE/AVIRmPHr/8QAGREAAwADAAAAAAAAAAAAAAAAAAERECFB/9oACAECAQE/AXaaHzH/xAAdEAACAgIDAQAAAAAAAAAAAAAAAQIRMUEhUWGR/9oACAEBAAY/Ar1dFxbNkpdSOHZgf0XphH//xAAcEAEAAgIDAQAAAAAAAAAAAAABABEhUTFBcZH/2gAIAQEAAT8hdJaGSDyw7NxOoAXu4myepjz9sxM4YXnC4zJPSf/aAAwDAQACAAMAAAAQsAe//8QAGBEBAQADAAAAAAAAAAAAAAAAAQAQESH/2gAIAQMBAT8QHFxNnbf/xAAYEQEAAwEAAAAAAAAAAAAAAAABABARIf/aAAgBAgEBPxBrVD2MJ//EABwQAQEAAwADAQAAAAAAAAAAAAERACExQVGB8f/aAAgBAQABPxAeZq9WVmcC80YLiXbfcFAVSC9AmMF6FjWIaILvbd795a6UQeU/MlrKKMsHZ0Fz/9k=","width":50,"height":50,"src":"/static/84204f0ce4111c333b6172133a845a85/d2d31/avatar.jpg","srcSet":"/static/84204f0ce4111c333b6172133a845a85/d2d31/avatar.jpg 1x,\n/static/84204f0ce4111c333b6172133a845a85/0b804/avatar.jpg 1.5x,\n/static/84204f0ce4111c333b6172133a845a85/753c3/avatar.jpg 2x,\n/static/84204f0ce4111c333b6172133a845a85/31ca8/avatar.jpg 3x"}}},"headerImage":null},"html":"The AUTO1 microservice landscape is growing in size day by day, with new services from a spectrum of languages being erected and usually communicating with each other by using REST contracts.
\nIn a fast paced environment, such as AUTO1, often the service providing the REST API and the service consuming this REST API are developed in parallel, by independent teams. On top of that the business requirements over the lifetime of a project evolve, thus impacting the stability of the API. Having these moving parts in picture can make the development process quite challenging, especially when aiming for high productivity, zero downtime and independent development of the client and server parts. Hence, having a clearly documented and language agnostic API is vital for the success of the whole process.
\nAfter extensive research, we found a great candidate seemingly accommodating all the aforementioned requirements - OpenAPI. In short the OpenAPI specification defines a language agnostic standard allowing to describe REST APIs in a human readable form, imposing a unified structure, clearly and comprehensively aligning expectations between the consumer and provider sides.
\nBelow is an example of such an API described using the OpenAPI 3.0 standard:
\nIn general, we distinguish two approaches when developing APIs. The code first approach, which is basically about developing the source code fulfilling the business requirements and afterwards, optionally, generating the API specification from it, which for OpenAPI can be done using the swagger annotations. The second approach known as design first, is all about describing the API in a human readable language first, by using for example the OpenAPI specification, and then optionally, based on this artifact generating the source code for server and/or client. Naturally, both approaches come with pros and cons.
\nAs previously mentioned, in AUTO1 it often happens that changes that are being introduced in the API on the server side need to be implemented at more or less the same time on the client side. In such a situation, if the code first approach would be practiced, the consumers would need to wait for the server side implementation to be delivered before starting their part, that would be simply inefficient. Here, for the rescue comes the design first approach, where from the moment when the API specification is agreed upon, both server and consumer side can kick-off their side of the implementation, independently. Additionally, this allows to spot misalignments already at a very early stage, and rectify them before the actual implementation from either of the sides is started, hence less expensive to fix.
\nThe OpenAPI comes with a rich ecosystem, from UI tools such as Swagger UI allowing to visualize and test ad-hoc the specifications to AWS integration providing a fast lane to expose the specification via AWS API Gateway.
\nAnother very helpful set of tools are the code generators, that can generate client side libraries and server side stubs from the OpenAPI definitions. These extendible generators support a variety of languages and frameworks, which lets one move faster, especially in a polyglot microservice environment. For our Spring Java applications we have decided to go for a generator that is very well configurable and easily customizable. It can be executed in various ways such as via cli, maven/gradle plugin, etc. Since our Java projects use maven for the build automation process, the maven openapi generator plugin was the natural choice for us.
\nDespite the vast configuration possibilities offered by the OpenAPI code generators, in order to integrate OpenAPI into our AUTO1 development world seamlessly and make the generated API and models classes adhere to our coding standards we needed to extend the default Spring generator that we’ve decided to go with in first place, to include some AUTO1 specific code structures.
\nTo be more productive, concise and spare some boilerplate code we use in AUTO1 the Lombok library, a lot, hence we couldn’t afford not having the Lombok annotations included in the generated classes as well. To follow the OpenAPI generator practices, we have added a configuration property to control enabling and disabling the usage of Lombok annotations:
\nThe above setting would result in generating the following annotations in the output classes:
\nSometimes the need arises to define custom annotations but only on some specific classes, for that purpose we have introduced a new attribute to our OpenAPI specifications called x-extra-annotation, that hints the generator to add additional annotations to the generated code. For example:
\nThe above attribute would result in generating the following annotations in the output classes:
\nGenerics is a very powerful feature that is available and widely used in the Java language for almost two decades now. Unfortunately, as of the time of writing of the article the OpenAPI does not support out of the box inheritance together with generics. To make it available to our developers we have introduced yet another attribute in the API specification named x-make-parent-generic specifying the genericType sub-attribute that indicates a generic type of the parent class (in this case PageDTO):
\nThe above attribute would result in generating the following generic parent in the output classes:
\nAs mentioned at the beginning of this article, our Java services are using the Spring Cloud stack with declarative Feign REST clients, originally the Spring Cloud Netflix Feign module was used for that purpose, however, with recent upgrades to newer Spring versions it was replaced by the Spring Cloud OpenFeign module that resulted in changes in the imported packages (import org.springframework.cloud.netflix.feign.* was replaced with import org.springframework.cloud.openfeign.*). So we ended up in a situation where services use either the old Feign clients (located in their respective packages) or the new ones, yet share the same OpenAPI specification. To support both cases we made it configurable for consumers to go with either one:
\nAnother shortcoming of the default OpenAPI Spring MVC generator was the lack of support for making the generated classes implement specific interfaces (using the “implements” keyword), as of version 4.3.1 only the “extends” keyword was supported. We have overcome this limitation by introducing a custom attribute x-codegen-implements allowing us to specify a number of interfaces the generated class should implement.
\nThe generated output for the above configuration can be found below:
\nTeams often run into the situation where they have to use multiple OpenAPI specifications to generate the necessary contracts to communicate with the upstream services. Unfortunately, the org.openapitools:openapi-generator-maven-plugin requires a separate configuration for each specification, which was a bit too verbose and too cumbersome for us.
\nIn order to circumvent this issue we have created our own maven plugin The main aim of it was to reduce the boilerplate configuration, yet keep the flexibility of the original plugin:
\nWhen dealing with a large number of microservices a common case is that they share common features, hence common models, within a single bounded context. In a typical Code First approach operating in the Java ecosystem this problem would be solved by sharing a common jar which would be then included by the interested parties. We were interested in achieving similar functionality but from the OpenAPI perspective. We achieved this by simply extracting part of the common functionality into separate specification files and referencing them from the individual OpenAPI specifications that would like to reuse these definitions.
\nThe challenge in having OpenAPI specifications referencing other OpenAPI specifications was the dependency management and versioning part. It was especially hard from the consumer perspective, since consumers would need to include both specifications in their respective versions (the referencing one and the referenced one) in order for the code generator to succeed. The solution for this was building API bundling, by adding to the specification a meta attribute called x-bundle-apis we could make all referenced definitions to be included in a single final artifact that was then uploaded in the CI pipeline to artifactory. This meant, from the consumer perspective, that they just needed to include a single dependency now, making their life significantly easier.
\nBy default, when bundling the API we’ve always included the latest referenced specification. Since we have had cases where we wanted to give more control to the API owners to specify which concrete version of a “common API” they would like to reference, we’ve included another attribute x-bundle-imports which could be used as following:
\nDespite the vast number of improvements we have already made in order to retrofit OpenAPI and its tooling into our existing AUTO1 ecosystem and practices, there is still room for improvements. Just to name a few. Although OpenAPI specification itself is language agnostic, we have language specific generators which in some cases need to be instructed on how to generate the output code, and unfortunately these generator hints often make sense only in context of a specific language. As an example, instructing the generator to make the generated Java classes implement some specific java interface. Since the OpenAPI specifications are enriched directly with these hints, it results in obscuring and cluttering the contract definition.
\nYet another challenge would be to generate more complex classes resembling the structure as seen when written manually in the times from before the OpenAPI approach was adopted.
\nAs presented in the article, OpenAPI is a very powerful tool that enabled our AUTO1 organization to move forward faster in a more structured and unified way in terms of defining and evolving the REST APIs. It’s not a silver bullet, as the design first approach and the OpenAPI specification come with a number of challenges by themselves. However, with the talented engineering force we have we were able to resolve most of them in an efficient manner, yet every organization planning to adopt this tool needs to weigh in the benefits and the shortcomings and make the call for themselves.
","fields":{"slug":"/openapi-journey/","tags":["auto1","engineering","openAPI"]}}}]}},"pageContext":{"slug":"/spring-5-indexer/","tags":["spring","spring-indexer","bootstrapping","performance"],"category":"Coding","author":"Mariusz Sondecki","date":"2019-02-05"}}