The PostgreSQL team announced recently a new release of the most advanced\nopen source relational database - PostgreSQL 12. As usual it comes with an impressive list of improvements\n(generated columns ♥️), one of them being long awaited by dozens of developers: improved Common Table Expressions.
\nI should first explain what are the Common Table Expressions for those who are unfamiliar with them:\nthe CTE’s, often called “WITH queries”, are SQL constructs giving a possibility of creating temporal data views for a sake of a query execution.
Essentially CTE is an additional query which results can be referenced in the subsequent CTE’s or the main query before which\nit is being placed. It should be clear enough with an example - here’s a sample query with two CTE’s taken from Postgres docs:
\nWITH regional_sales AS (\n SELECT region, SUM(amount) AS total_sales\n FROM orders\n GROUP BY region\n), top_regions AS (\n SELECT region\n FROM regional_sales\n WHERE total_sales > (SELECT SUM(total_sales)/10 FROM regional_sales)\n)\nSELECT region,\n product,\n SUM(quantity) AS product_units,\n SUM(amount) AS product_sales\nFROM orders\nWHERE region IN (SELECT region FROM top_regions)\nGROUP BY region, product;Now we know what they are, but what purpose can they serve us? Well - we could parry here and say: for the same purpose as ordinary database views serve.\nThat’s of course a dramatic simplification - CTE’s are much more powerful and should not be treated as a simple database views.\nNevertheless I would like to keep the collation for the sake of this article.
\nDatabase views are absolutely optional - one can simply substitute them with a subquery and achieve identical results.\nIndeed that’s what modern database engines do these days - once a database view is being used they inline it’s query as a subquery.\nWhy to bother then? We are able to deal completely without database views and even if we did use one the database engine would get rid of it anyway.
\nWhat benefits do views give us then? And why the heck are they inlined?
\nThe most appropriate explanation here is that database views help us achieve better readability.\nThey offer an elegant way of abstracting some parts of a database into a meaningful object, often matching closely with the domain.
\nInstead of creating giant and ugly looking queries it is possible to extract some of its parts into an appealing view which is easier to browse and select data from. \"Divide and conquer\" rule in it’s true form.
\nStill, we didn’t answer the fact that the underlying view’s query is most of the times inlined while it is being referenced. The reason is performance of course. Smart guys found out that lazy evaluation helps the optimizer a lot - by delaying the execution we could take advantage of a context of the actual query.
\nThis in turn allows many clever optimization techniques, like: pushing down predicates (WHERE filters), eliminating unnecessary JOINS, accessing only subset of columns etc.\nIn other words - a database is smart enough to do as little work as possible when evaluating database views, in the context of the issued query.
\nPersonally I love this pattern: aggregating all the data into views and letting a database to optimize my queries - these folks are really good in it and my queries are dead simple too.
\nI have mentioned the CTE’s at the beginning, saying they are able to create temporal data views. I still conform to the comparison with database views - they both are in many cases similar.\nThe main difference is that CTE results are temporary and are reachable only in the context of a query which CTE is being part of.
\nIt may make sense to use a CTE in a place where a regular database view is not justified (e.g. it makes sense only in the context of a query and not in the whole domain), expecting similar behavior.
\nBesides many remarkable advantages there’s at least one disadvantage which disqualifies CTE's for most use cases - before PostgreSQL 12 it was implemented as an optimization fence. What does it mean?
\nEasy to imagine an example with a generic data view aggregating lots of data. If the view is then used to select just few records it could mean a tremendous waste of computation, if the aggregation is executed immediately.\nInstead the aggregation should be executed only on a small subset of data, which can be deduced from the outer query.
\nUnfortunately such counter intuitive behavior was true for CTE’s for a long time - their results were materialized only to be accessible for the rest of the query afterwards.
\nNot to blame anybody - the creators had quite good reasons to implement such behavior (i.e. guarantee of exactly one evaluation, possibility of a recursive CTE’s and more) but this still feels like focusing on corner cases instead of optimizing the happy path.\nThat’s why the community insisted for a long time for changing the status quo by giving the possibility of disabling the fence and unlocking the full potential of CTE’s.
\nThe SQL gods listen to their prayers and here it is - PostgreSQL 12 with updated Common Table Expressions.\nBy default, when few constraints are met, the queries will be inlined allowing joint optimizations.
\nIt is still possible to force the old behavior - by defining the CTE AS MATERIALIZED the engine would execute it immediately.\nIt is also possible to hint the optimizer that we definitely want the CTE to be inlined, e.g. when the CTE is being referenced twice the engine won't inline it by default.
This is truly a game changer for many developers who care about their queries’ readability, allowing them to substitute not very well liked subqueries with elegant “WITH queries”.
Don’t get me wrong - not every subquery should be immediately replaced, they still have their strengths and in some cases they should be chosen over CTE’s. It is just convenient to have two distinct tools in a toolbox, isn’t it?
","fields":{"slug":"/postgres12-a-precious-release/","tags":["postgres","postgres12","release","sql","cte"]}},"categoryArticles":{"edges":[{"node":{"id":"f3a0fb4b-eb2f-5624-a408-d63e3cee5184","frontmatter":{"category":"Coding","title":"Spring batch integration","date":"2020-02-07","summary":"Usage cases for spring-batch.","thumbnail":null,"authorName":"Artur Yolchyan","authorDescription":"Artur is a Senior Software Engineer at AUTO1 Group.","authorAvatar":{"relativePath":"pages/spring-batch-integration/avatar.png","childImageSharp":{"resolutions":{"base64":"data:image/png;base64,iVBORw0KGgoAAAANSUhEUgAAABQAAAAUCAIAAAAC64paAAAACXBIWXMAAAsSAAALEgHS3X78AAAEfElEQVQ4yz2O2VMbBQDGlzJQSylXIEc3F0eODYGEJAQI5GBzbK49kt0cm703CSRQqfQCB9taap1qgY61VXqoPZxxbKe2dWo7tY6d0RdHX/0HfPLRF991qTPO/OZ7+33fB+RDFjruKMWhcgwqwnYqaiVhKwXbiolRgZiuVxJ1LrsgEg25sFilFqrFBblQFXNVgajyBMCkxiuIs4yMMoiDSTiwoC3kc/hdVpdVPzFqRua9MpupS7mqmK9LVF0u1CVFzssCIXMYwKZcTGpMyWLUOe2yjoxAQyNO1YCutbV9f1tbe1u7dcRY47GaMiXuVVSFvMTjModLLApwaRefHqeibqNm4EBnb/+A1mo2T9kHcZ9tGR5LQocPvdFmNusEOqv8lPicxBESi4kstifzabeMuj1Wnba7i/Q71vDgJ/Xc4zXhyWrxq1pyG/eWvEMd+9umfWMLAiFWshKD/g8goROV5LjdBJa9ls9r2edr4svTjRenG8836o9WK7e46EXM5zbpNGoVX0qKdEqk00qFgkBnAGrOko+Nz1jA94iZhyvFZ8fZ707JLzaWft48/vJ08/6b5Y+K4azHph5QlbCIXE7yJYRXco8U4Ffvn7ZoAiPgRSp8vU59s9Z89s7RJ2fXtxerbxGpo5nIOWxaCLr1BhOZDUqlOF+ICyVEQSwnAcuBFmtHi8ekRT0OMjBZQyI3lvgduVCLzvGx0HJi9nw+WPTbJ4dNdSLMUTBXiAnF+GsSgKN7n62zxWnQVAIuOjCxEJnaxuevleMfkMjJzPxGcuZTPs4FoA0U3uHIIhHkqKgCr1CIAx6wBeptTY3ZTiRnzmPhbTp9gyFuc+hNBnm6Kj48tXy1Ri7Brs8WxTMVvITN8fkIT8I8GRXIKFCAR11g1wSo3WSIb9dXbojEnUb1C5m5XkJ+ePvI4/WVHSm3BE98SKNCZpbBg6xyntzz92QxN+U39XYBQDMdfbTWvCpiF/Kpyyy1Wy/fapS+XC5tcclq0JGdMOViHhYP/Sdzyn5+HuBR3+xwXwcABG1DNxvkOgqfLxR3q4vvoujFQvzeSuGymKV8w5C+m4p7GTTIYEE2F+b2iABcZnLW2tsNAPrOg5f4zJVyYgfH7vHibY76epm8eyR/BpuU0inPNBFwW6RcSPGVfQYPcbl5gM14Ex69/gDQvW8f7rHuyujtGnm3yT84Id45Sp8lw0dwZHPrqX0s3N/Xnwl56UyAI0KvCQN02lOIOcfV7erWFvBgOwwZj+VSm0LleDlfR+aOsfSDV3+yS1udHV2aAZNaBQ4aByddTjQ6VcGCAJv1wX472K9S96gOHeyZmkHoWJiJ+Jsxz9ax1Ve//33p3k86tVmnM2s0JgWtdlCrHdKDw5AVAoio+7DW2NOj6+lRazQj71/99crayY8b+P3rd3/5459Hv/017o7otENGo82gt5jNo0rq9cMGg0UPWoBZj7O3Dxzo1/f1alyT6OL645Xm2cvXflw+93Rz93tx6YJaZRocHDMYIKMRAg9bQHDYZIIMBrvZ7PwXcPlkRtKKCQsAAAAASUVORK5CYII=","width":50,"height":50,"src":"/static/dc9516b9327e627890a1f5c7f84c004a/45876/avatar.png","srcSet":"/static/dc9516b9327e627890a1f5c7f84c004a/45876/avatar.png 1x,\n/static/dc9516b9327e627890a1f5c7f84c004a/eb85b/avatar.png 1.5x,\n/static/dc9516b9327e627890a1f5c7f84c004a/4f71c/avatar.png 2x,\n/static/dc9516b9327e627890a1f5c7f84c004a/9ec3e/avatar.png 3x"}}},"headerImage":null},"html":"Intro
\nIn this article, we will take a look at spring-batch and how it could be used. We will walk through various configurations and we will create an application which reads from a CSV file & writes into a database with outstanding performance.
\nI used the following: Java 11, Spring 5+, Spring Boot 2 and Maven-based project.
\nCreate a Project
\nFirst, we need to create a Spring Boot 2 project. We recommend that you do so by visiting spring initializer, which is a useful tool to generate spring projects with required dependencies and configurations.
\nDependencies
\nWe need the dependencies below to run and test the project:
\n<dependency> \n <groupId>org.springframework.boot</groupId>\n <artifactId>spring-boot-starter</artifactId>\n</dependency>\n \n<dependency>\n <groupId>org.springframework.boot</groupId>\n <artifactId>spring-boot-starter-test</artifactId>\n <scope>test</scope>\n <exclusions>\n <exclusion>\n <groupId>org.junit.vintage</groupId>\n <artifactId>junit-vintage-engine</artifactId>\n </exclusion>\n </exclusions>\n</dependency>\n\n<dependency>\n <groupId>org.springframework.boot</groupId>\n <artifactId>spring-boot-starter-batch</artifactId>\n</dependency>\n\n<dependency>\n <groupId>org.springframework.boot</groupId>\n <artifactId>spring-boot-starter-data-jpa</artifactId>\n</dependency> \n\n<dependency>\n <groupId>org.projectlombok</groupId>\n <artifactId>lombok</artifactId>\n <version>1.18.10</version>\n <scope>provided</scope>\n</dependency>\n\n<dependency>\n <groupId>com.h2database</groupId>\n <artifactId>h2</artifactId>\n</dependency>\n \n<dependency>\n <groupId>org.springframework.boot</groupId>\n <artifactId>spring-boot-starter-test</artifactId>\n <scope>test</scope>\n <exclusions>\n <exclusion>\n <groupId>org.junit.vintage</groupId>\n <artifactId>junit-vintage-engine</artifactId>\n </exclusion>\n </exclusions>\n</dependency>\n\n<dependency>\n <groupId>org.springframework.batch</groupId>\n <artifactId>spring-batch-test</artifactId>\n <scope>test</scope>\n</dependency>\n\n<dependency>\n <groupId>org.hamcrest</groupId>\n <artifactId>hamcrest-all</artifactId>\n <version>1.3</version>\n <scope>test</scope>\n</dependency>spring-boot-starter-batch dependency includes all the configurations to run the spring batch application. Lombok is just a helper dependency to write the code faster and cleaner. H2 is used as an in-memory database. spring-boot-starter-test and spring-batch-test are included for test purposes.
\nBook Class
\nLet’s create a model class book, which will represent a book. This will just serve as a model class and will help us during implementation of spring batch.
\n@Data\n\npublic class Book {\nprivate String title;\nprivate String description;\nprivate String author;\n}Configuration
\nLet’s create a class called SpringBatchConfiguration. We will add all required configurations here. First, let’s annotate this class with @Configuration to be able to inject beans, and with @EnableBatchProcessing to enable spring batch processing. Additionally, we can add @RequiredArgsConstructor from Lombok which would help us to generate constructor with parameters. These parameters are the ones which are marked as final class properties. These properties will be injected as spring beans. Our class will be like this:
\n@Configuration\n@EnableBatchProcessing\n@RequiredArgsConstructor\npublic class SpringBatchConfigurationNow, in SpringBatchConfiguration class let’s add properties which need to be injected into the constructor:
\nprivate final JobBuilderFactory jobBuilderFactory;\nprivate final StepBuilderFactory stepBuilderFactory;jobBuilderFactory and stepBuilderFactory are declared in spring batch jar as spring beans so that we can inject them in any class we want.
\nFile Reader
\nNow, we need to declare and initialize spring beans to configure the batch process. The first bean which we will need will be responsible for reading from a file line by line. Spring Batch provides a default class for it. The class name is FlatFileItemReader. Similarly, spring has different default reader classes for reading from a relational database, mongodb and etc. However, if you need, you can create your own reader and implement it in a way you want.
\nLet’s now see what FlatFileItemReader injection will look like.
\n@Bean\n@StepScope\npublic FlatFileItemReader<Book> bookReader(@Value(\"#{jobParameters['filePath']}\") final String filePath\n) {\nreturn new FlatFileItemReaderBuilder<Book>()\n .name(\"personItemReader\")\n .resource(new ClassPathResource(filePath))\n .delimited()\n .names(new String[]{\"title\", \"description\", \"author\"})\n .fieldSetMapper(new BeanWrapperFieldSetMapper<Book>() {{\n setTargetType(Book.class);\n }})\n .build();\n}We are configuring that the bean reader should read data from the given file path, which should be a CSV file having rows with title, description and author respectively.
\nThe StepScope means to initialize this bean after each step, so file path could be dynamically set when the spring batch job is launched. We will come to that later, how to pass the file path later in this article.
\nItem Writer
\nNow, let’s create a writer class, which will take the data and write into the relational database.
\n@Bean\npublic JdbcBatchItemWriter<Book> writer(final DataSource dataSource) {\n return new JdbcBatchItemWriterBuilder<Book>()\n .itemPreparedStatementSetter((book, preparedStatement) -> {\n preparedStatement.setString(1, book.getTitle());\n preparedStatement.setString(2, book.getDescription());\n preparedStatement.setString(3, book.getAuthor());\n })\n .sql(\"INSERT INTO books (title, description, author) VALUES (title, description, author_surname)\")\n .dataSource(dataSource)\n .build();\n} In the writer bean, we are setting item processors and adding values inside preparedStatement accordingly, which book property should be inserted for each db table column.
\nStep Configuration
\nNow, let’s configure a step which will be executed in the batch process. In our case, the configuration will look like this:
\n@Bean\npublic Step step1(final ItemWriter<Book> writer, final ItemReader<Book> reader) {\n return stepBuilderFactory.get(\"step1\")\n .<Book, Book> chunk(100)\n .reader(reader)\n .processor((ItemProcessor<Book, Book>) book -> book)\n .writer(writer)\n .build();\n}In our scenario, we have only one step, but it's also possible to configure multiple steps. Here, we are creating a spring batch step with the name step1 and setting reader and writer accordingly. These are the readers and writers which we created as spring beans earlier.
\nWe are setting chunk as 100, which means the items chunk proceeded is 100. We can make this configurable too. The processor is for converting the current object to the one which the writer should proceed with. In our case, it is the same object.
\nJob Configuration
\nLast but not least, let’s configure the job which should be executed.
\n@Bean\npublic Job importUserJob(final Step step1) {\n return jobBuilderFactory.get(\"bookReaderJob\")\n .incrementer(new RunIdIncrementer())\n .flow(step1)\n .end()\n .build();\n}Here, we create a job with the name bookReaderJob. We add an incrementer RunIdIncrementer, which is an id generator for the tables which are specifically designed to store the details about job executions in the database. After each time the spring batch job is executed, it will save details about the execution. To check the schema structure for storing this data, take a look at this sql. It is for MySql, but other SQL scripts are available too.
\nAdditionally, we add flow by which the job should be executed. Currently, we have only one step in our flow, so we add it.
\nPlease also add this config: spring.batch.job.enabled=false in your properties config file so that spring batch job won’t be executed automatically with no parameters when the application is started.
Execution
\nTo execute the job, we need to declare JobLauncher and launch the job. To do so, we need to create the below class:
\n@Component\n@RequiredArgsConstructor\npublic class SpringBatchExecutor { \n private final JobLauncher jobLauncher;\n private final Job job; \n\n @SneakyThrows\n public void execute(final String filePath) {\n JobParameters parameters = new JobParametersBuilder()\n .addString(\"filePath\", filePath)\n .toJobParameters(); \n jobLauncher.run(job, parameters);\n } \n}Now, we can call the execute method from whenever we want, with the file path from which the data should be read.
\nAdditional Classes
\nThere are a few additional classes which you will need to execute your code.
\nCreate a class BookEntity so that spring data JPA will automatically create the book table for you. Then you can create repository.
\n@Data\n@Entity\n@Table(name = \"books\")\npublic class BookEntity {\n @Id\n @GeneratedValue(strategy = GenerationType.IDENTITY)\n private Long id;\n private String title;\n private String description;\n private String authorFullName;\n}Then create the interface BookRepository and extend it from JpaRepository. At the moment, we will need this only for testing.
\n@Repository\npublic interface BookRepository extends JpaRepository<BookEntity, Long> {\n} Testing
\nNo one likes a code which is not tested. So, let’s write a few test cases for our class.
\nHere is a code sample for test cases:
\n@Slf4j\n\n@SpringBootTest\nclass SpringBatchSampleApplicationIntegrationTest {\n\n @Autowired\n private SpringBatchExecutor springBatchExecutor;\n \n @Autowired\n private BookRepository bookRepository;\n\n @Test\n public void testExecution() {\n long initialCount = bookRepository.count();\n assertThat(initialCount, equalTo(0L));\n springBatchExecutor.execute(\"sample-data.csv\");\n long count = bookRepository.count();\n assertThat(count, equalTo(7L));\n } \n\n @Test\n public void testLargeData() {\n long startTime = System.currentTimeMillis();\n long initialCount = bookRepository.count();\n assertThat(initialCount, equalTo(0L));\n springBatchExecutor.execute(\"large-data.csv\");\n long count = bookRepository.count();\n assertThat(count, equalTo(60000L));\n long endTime = System.currentTimeMillis();\n log.info(\"executed in miles: {}\", endTime - startTime);\n }\n}The second test executes in ≈ 3500 ms in a machine with a 2.2 Ghz Intel Core i7 and 16GB ram. It proceeds 60K lines of CSV file, and saves it into a relational database.
\nYou can check out the working sample code in github,
","fields":{"slug":"/spring-batch-integration/","tags":["auto1","engineering","spring","code","java"]}}},{"node":{"id":"a3a21c38-873e-56db-a170-eec0e6136040","frontmatter":{"category":"Coding","title":"Writing IntelliJ plugins","date":"2019-12-10","summary":"How to write a simple IntelliJ plugin","thumbnail":{"relativePath":"pages/intellij-plugin/logo.png","childImageSharp":{"resolutions":{"base64":"data:image/png;base64,iVBORw0KGgoAAAANSUhEUgAAABQAAAAUCAYAAACNiR0NAAAACXBIWXMAAAsSAAALEgHS3X78AAAEyElEQVQ4y3WVe0xTVxzHz723rfZBsVeqnc4Yw0aYFSiFFZlZxnSo4w+gSBWEglTmA59jCt4WCOUxq+J4zGwaMqfJfDBi2GbMxjJfMXMzJks0Y5ngfE0mqAwrFLHtPb+de4CKf+wkv5zfOed3Pr/vOb+Te9FAzEokNV9cLvq/Nmpew8L8DIQOANJUPEdh+wGpBB8jrSldo0hd7n0RLJry0fA4bCg2RxYw5S3zm/ISnsWt1vvi8qZMxD1KLkVvb+lmJifS7hrmpJ4lCfidAy8WAqZ8dqzPSwFzIQRN+UEy95SMbxK7MhK3umQiVlnuLVQKwxcVW//SS2O94JNFFZyja3NLHyD03JSHIN5OMxHQZ5C4Fgg0CAlFAIkOAMs6gIWbiV9sQ0cgTlMl+tW7ATSV+E64y2+R9kWVDXJR226PZZSAUvPG5mpxvP0+MfjjjfTg9eh0/Nv8TPGXBdmBS3H5cC52tS8253SffNMN0BZfHdVu7AK+tC9gIIon1MdsvkXvkKojd2aT1N02ZgXDWBnmGBbLJWM5IL4oZzhQEEOMTESsHBBiAjNzz8BrVSJEb721b3taOzdWlIQien+kECfB8gHciM8NkCHOyLLC711d+MPSUkzGIM159njE5IVJ1CfVgXnZbeKbpQ9Ey84+iCy7kxUqylB8QeQoUTeatAWumteIMhKs43m42dMDFotFAgDDMHD+/HlIT88YA7IsNmUcwWmbb4hRu+4BX9mfGQI+SSiY/SBxneeflJLHZ5NKQM7IqKofOzth1qxZMK4QOjo6CDCd+izLwNLlLXhh+R2sr3oEM4SHy8bUOYz07H0daFrQNWPw57XJQG8iIQEwxmC1WiE1NRWampqgu7sbzGYzBUqniLV9gee4B3GE6zHonAMpFAh7ZPSx+sqmH4Pa6XB5++vSHcJbixbhpsZGSEtLA6PRCK2trZCTkxNSK5m+oB2/4h6WYDjcOZgcOvLDHbxyZJfuT6ifDhc3zA8qZIqXNr5sTMiPsJ/CevcoEFgwzPnETGHPBB1VeG+TwQDV2rNnC98FBmlEhUKG1So1TJ06FeRyOchkMuA47iWgzv4t5msCQGB+ldNrDCns3b6AQgGOs42LS67PnR0J+z9pCDY3N+ODhw7i9RvW08qOW8jX2k/jaTUY1M6nI0rhaSSF9e2IRlXvXKD+paJrEbVL9varp6hBGx7m53lejNDxYrhKE6TKGAYj8tiJT4Ea+5mA1o1B5Rz2KwXvnJDClszDtNKfWw9bW7KOwrYl9bAltQGKF3sgb/k+WJpzAGIcXwYiHV/j2Y7vxAjHD4FwxznQurygqQdQunw9yrIBLYV5rEdRQ/YJCty74qujhwp/gt253/R/bP/+eJPj8vvr3JdWvld95X6Spxeiq//Gc2uegMEDoHP7Iaxy5ILKOVREjjtN2k+UIlS34hgF12UeVtRln/i02ta2aWN6q37ydy9mfQdvFK61zqu4CwZnbx8v9Ddrd95PnBxDVI455dknaE9A7OSA+lWn2Orsdi7zo05uYu5V513LTKHXMDHW12Gkcvk4pTBEi6qegLptJ2nfU9CJam1tXKWtjRFWtSND1S06b3H+yiRWXg8l1Ff8y5G3x6oELyJK0cwa6TfwmK79B0KrRX19cxThAAAAAElFTkSuQmCC","width":325,"height":325,"src":"/static/b2f976ed5fa752977d8f8f9b2516534c/b3029/logo.png","srcSet":"/static/b2f976ed5fa752977d8f8f9b2516534c/b3029/logo.png 1x,\n/static/b2f976ed5fa752977d8f8f9b2516534c/8d141/logo.png 1.5x,\n/static/b2f976ed5fa752977d8f8f9b2516534c/ee72c/logo.png 2x,\n/static/b2f976ed5fa752977d8f8f9b2516534c/5dfa8/logo.png 3x"}}},"authorName":"Piotr Czekaj","authorDescription":"Piotr is a Senior Software Engineer at AUTO1 Group.","authorAvatar":{"relativePath":"pages/intellij-plugin/avatar.png","childImageSharp":{"resolutions":{"base64":"data:image/png;base64,iVBORw0KGgoAAAANSUhEUgAAABQAAAAUCAYAAACNiR0NAAAACXBIWXMAAAsSAAALEgHS3X78AAACyElEQVQ4y4VUO2gUURR9mZmN8YcItoKVjeXuzsxu9jOzm92NkBBTqDEmjZAEgtqIpSgI1gp+uhS6kEI7wc7GQsFCLERiISGCaGHhF02yu+M5s/cujyC4cJY379573r3n3XdNvV51xsYiJ4rKbqlUyBj8wjAYyWazp4BVYB3YAraBDeAhMBcE/l76lkrFTLVachuNyCGXEbJMrVbx6ADnaWAtn88nvu8nWBMdQcJ9Auv3wCxjWq2aC9IMSF3DzCyyG1YAs9pSYkL2melmLpfT71tKGkUlz2iZJAuCgA5dEjGAwPoV0AbuAy+tfR7YkZjb5CiXR1OutEw5jWQdCfgCnEiSxNg/7LWAj+KTSiHSzBvrAtascrriPCmnDkP4AYQ0Em3TGPHf8P38ARpP/4PsHQ7aU69Hqdi4OJeI47JXLg9IX1tVbUuWCzSsygcJO+L0goGNRpySoB1colarerhEvcCn4tuxCB/TsC5ZdS3Ct2Ho75uZmRoigWbIbgCxt7Q0O8TLsgg17o2R2+JHzzJcV/1IoBlybel42SLsCcc3I6UqoaZ+lUHFYjjCzJSQa+4J4UVLKiX8ScOGVXJXDN+xd7SfZXFYS65U+n2Wz+cOw+ezHadSkfCRdVJiZdlmMG9ZtHMpgWR3Z0eMSvWExjmrbXoCbZ951dIim7Yk6u1IYtlwavChW51Pp01tXOwf01eC9RHs/Rbbnx2Hf4IUh/Q5zdol8H3KG22Pj9edOK54LF1876pd37TELhqdZ+J4MwxDHUvXgDOFQrB7YqLpjI4WdhG8GLYNbCeBK/1+TWNWRG+Pfy5Hj5Dek3IXzX9+8DkL/AIeLC/PDU1NHXdSQk5altNsDkjPAT+ArzKyFnK5bB2IxbYik4hanmfM5GTL6Q/Y2DUc25y0HLToubR8TI39cL4APJOxr/35AXgOXEJnHKQvepP6eiQj119kDrZjqLo8HAAAAABJRU5ErkJggg==","width":50,"height":50,"src":"/static/26a7a327ccc4b335712e5a7086f2b26d/45876/avatar.png","srcSet":"/static/26a7a327ccc4b335712e5a7086f2b26d/45876/avatar.png 1x,\n/static/26a7a327ccc4b335712e5a7086f2b26d/eb85b/avatar.png 1.5x,\n/static/26a7a327ccc4b335712e5a7086f2b26d/4f71c/avatar.png 2x,\n/static/26a7a327ccc4b335712e5a7086f2b26d/9ec3e/avatar.png 3x"}}},"headerImage":{"relativePath":"pages/intellij-plugin/header.png","childImageSharp":{"resolutions":{"base64":"data:image/png;base64,iVBORw0KGgoAAAANSUhEUgAAABQAAAALCAYAAAB/Ca1DAAAACXBIWXMAAAsSAAALEgHS3X78AAADGklEQVQozx3R60/bVRjA8V80m2N4IVkWcfpmqyulTdd7oRYopEDbyaXlVgptobANkAzrUpwbyjZvgMpQ7CxhGCaXUTcv04mRRYMkKpB1MLYpIKkymb5S/4WvZ7745DkvnnOey5FS87qQWbqY0zWyIEzrmxkxHOFHcXZkRZCsZ0ixnmKH9TQPWV7m0eIo22q/JmjvI6ENsijyh7I62G7rYXt2F1KgpIo2TZhlbYBFkTCjDaE2Rdhn6mS36TiPC+nml9ilP0a6rQ91wxwZhxJ4ar9kMbuDdeMR+h0DyAOzyBwxpJDhMCsHAlzXBFgQ7mj8DGqaeUr3Agrt8yh0YWQiZolOq9wTuLxXcFZ/SmHNFVrKL/DKwXMcrPkcu2+aYkGKqkKsquu5rvZzQ7gf19R1nFTUo1M0Y5E3YNWGidu6mSzsQdc0j+ZwgrzgLFUVcWEKn/uiMInfPY40pPKxrvWypPFyU7ilqWZJX8dr9l4UrjFUzlFy3OJS8BqWtmVkRzeQtf/K3qNJCsT4EWeUU65+ely9HHOdQ3pR4Wa1OI8Vez637DaWiwp5tfxtIidmGbywxsilJH2xn3lneI3RyXU+iK3Q1votPu9VfP5vOOGMEbeHmXEcoregV+wwvYx4volk816SoSeZrS3lfPmbfFj2OqPeQcbqo4z7+pnwDvBxYJjJ2iFGSt8j5jhLzPUuY7ndXLREmMruZMJyHCmyq4jO3UWMGg1MO+XM5z7LkiHEDwcamVM28JM6wFpODas5Xm7omljRt3DH1Mptcxu/GFv4yNlNZ3CQPk8P14ytSG+l5XHmESund+ZzcmcuX+xx8ldmNckMD78pKthUevjHXMK/5lLuZlYIlWwpq9lSVJJUeQkFL2N77ntsHfO8URpFGk41M572DLFUI++nGDj/sJnb+0q4l1nFH8LW/yrZ2O8mKfdwVxT5/X4x4c8MN+3ip1XtGxj8X+GouYw081gW9+TlfJaiY/xBJWMPKPlkh5bEEwXc3GMnkV7Ad2nZxLepuZpqYHN/GVvisU3h76dLmDKGsTQuoK+7RJNrgP8AU8rtTZmilCkAAAAASUVORK5CYII=","width":1280,"height":720,"src":"/static/6fae10ea7503a11a8f308ac2e2a5217a/26421/header.png","srcSet":"/static/6fae10ea7503a11a8f308ac2e2a5217a/26421/header.png 1x"}}}},"html":"During analysis of why some services take a long time to build, several problems have been found, mostly related to number of Spring contexts that were created during integration tests. More about those problems can be found in previous blog post. In order to decrease the likelihood that slow tests will appear at AUTO1, we implemented an IntelliJ plugin which warns a developer about possible problems when he writes code and provides quick fixes if they are applicable.
\n@DirtiesContext annotation which slows tests down because it forces creation of new Spring context@ActiveProfiles (slows tests down because different Spring context would have to be created for each combination)@ActiveProfiles is greater than 3@FeignClient (or DTO used by client) not documented with Swagger annotations@TransactionalDuring the plugin development, we realized that resources on writing IntelliJ plugins are scarce, and therefore decided to create a step by step tutorial on how to implement @DirtiesContext inspection.
In our tutorial we will be using the community edition of IntelliJ 2018.3.5. Final source code is available on github.\nThere are several ways on how to create IntelliJ plugin. The recommended one is to use Intellij plugin for Gradle and that’s what we will use in this tutorial. Let’s start with creating new project using File > New > Project..., select Gradle and make sure that in Additional Libraries and Frameworks both Java and Intellij Platform Plugin are selected.\nIf you cannot see “Gradle” or “Intellij Platform Plugin” then please make sure that both “Gradle” and “Plugin DevKit” IntelliJ plugins are installed.\nContinue next steps of the wizard using default values, use any groupId and artifactId. After some time IntelliJ will download dependencies and create an empty plugin.
\nOur inspection should warn a developer each time there is a class annotated with org.springframework.test.annotation.DirtiesContext and provide quick fix which deletes @DirtiesContext. If Spring is not your thing then any other annotation can be used instead.
There are two kinds of inspections:
\nLocal inspections are executed in the background when file is opened, in general they have access to currently open file. Local inspection cannot report problem for not currently processed file. Inspection class has to extend com.intellij.codeInspection.LocalInspectionTool or one of subclasses.\nGlobal inspections work only in batch mode when analysis is manually triggered via Analyze > Inspect Code and see complete graph of references between classes and can report problem for any file. Inspection class has to extend com.intellij.codeInspection.GlobalInspectionTool or one of subclasses.
We don’t need to access complete graph of references and we would like to give a hint that something is wrong as soon as possible so local inspection is a better choice for our use case. First problem that we encounter is to pick proper base class. One approach is to ask IntelliJ to show class Hierarchy of LocalInspectionTool and take a look what other inspections are extending. In this case AbstractBaseJavaLocalInspectionTool seems to be a good choice since most of Java inspections are based on it.
Create new class named DirtiesContextInspection with following content:
public class DirtiesContextFirstVersionInspection extends AbstractBaseJavaLocalInspectionTool {\n private static final String DIRTIES_CONTEXT = \"org.springframework.test.annotation.DirtiesContext\";\n\n private static final String DESCRIPTION_TEMPLATE = \"Usage of @DirtiesContext makes integration tests slower\";\n\n public String getDisplayName() {\n return \"Usage of @DirtiesContext is not recommended\";\n }\n\n public String getGroupDisplayName() {\n return GroupNames.PERFORMANCE_GROUP_NAME;\n }\n\n public String getShortName() {\n return \"DirtiesContext\";\n }\n\n public boolean isEnabledByDefault() {\n return true;\n }\n\n public PsiElementVisitor buildVisitor(@NotNull final ProblemsHolder holder, \n boolean isOnTheFly) {\n return new JavaElementVisitor() {\n\n public void visitAnnotation(PsiAnnotation annotation) {\n super.visitAnnotation(annotation);\n\n String qualifiedName = annotation.getQualifiedName();\n\n if (DIRTIES_CONTEXT.equals(qualifiedName)) {\n holder.registerProblem(annotation, DESCRIPTION_TEMPLATE);\n }\n }\n };\n }\n}The most interesting thing happens in a visitor which is notified each time a Java annotation is encountered in source code. By overriding different methods, plugin can process methods, classes, imports etc. In our case we check if fully qualified name of annotation matches our expectations and register a problem when that’s the case. Later on we will change this class to include also a quick fix. Creation of inspection class is not enough to make it available in IntelliJ - it’s also needed to register inspection in plugin.xml which among other things describes what plugin does, what other plugins are required and in which version of IntelliJ it can be used.
It’s possible to register each inspection one by one in plugin.xml under extensions tag and configure inspection using xml but we find it easier to register inspectionToolProvider and configure using it in Java code.
Create the following class to implement our inspection provider:
\npublic class CodeInspectionProvider implements InspectionToolProvider {\n public Class[] getInspectionClasses() {\n return new Class[]{\n DirtiesContextInspection.class\n };\n }\n}And register it in plugin.xml:
<extensions defaultExtensionNs=\"com.intellij\">\n <inspectionToolProvider implementation=\"com.auto1.intellij.tutorial.CodeInspectionProvider\"/>\n </extensions>Now it’s time to check our inspection in action by running runIde gradle task. This could be done from terminal by executing ./gradlew runIde but it’s better to use Gradle view in IntelliJ since it allows to start IDE in debug mode (if needed just right click on task and select Debug, you can create run configuration to speed-up in the future). New IntelliJ instance should show up, if you already have sources of some project that uses Spring Boot then open it, otherwise you could create a new project. Annotate some class with @DirtiesContext annotation and observe inspection marker showing up. In case of problems logs can be found at build/idea-sandbox/system/log/.
Many inspections report not only problems but also provide automatic ways of fixing issues. In case of DirtiesContext, it’s not possible to provide safe way of removing it because DirtiesContext is often used when bean holds some state which makes tests dependent on each other. Usually we want to remove the annotation and then figure out the \"dirty parts\" and clean them up in an elegant way. Since the second part is hard to automate we will provide quick fix which only removes annotation.
\nGo back to DirtiesContextInspection class and add quick fix:
private static class DeleteQuickFix implements LocalQuickFix {\n public String getName() {\n return \"Removes usage of @DirtiesContext\";\n }\n\n public void applyFix(@NotNull Project project, @NotNull ProblemDescriptor descriptor) {\n descriptor.getPsiElement().delete();\n }\n\n public String getFamilyName() {\n return getName();\n }\n }Next step is to pass quick fix when problem is registered:
\nholder.registerProblem(annotation, DESCRIPTION_TEMPLATE, new DeleteQuickFix());Executing runIde Gradle task should prove that quick fix works as expected.
Up to this point we have used hardcoded strings inside inspection name and description. To allow the plugin to be accessible in different languages we can externalize the messages. For this purpose we can use properties file. First create PluginBundle class:
public class PluginBundle {\n private static Reference<ResourceBundle> ourBundle;\n\n private static final String BUNDLE = \"com.auto1.intellij.tutorial.PluginBundle\";\n\n private PluginBundle() { }\n\n public static String message(@NotNull @PropertyKey(resourceBundle = BUNDLE) String key, \n @NotNull Object... params) {\n return CommonBundle.message(getBundle(), key, params);\n }\n\n private static ResourceBundle getBundle() {\n ResourceBundle bundle = com.intellij.reference.SoftReference.dereference(ourBundle);\n if (bundle == null) {\n bundle = ResourceBundle.getBundle(BUNDLE);\n ourBundle = new SoftReference<>(bundle);\n }\n return bundle;\n }\n}Then create a new file matching BUNDLE constant, in my case it will be src/main/resources/com/auto1/intellij/tutorial/PluginBundle.properties with content:
inspection.dirties.context.display.name=Usage of @DirtiesContext is not recommended\ninspection.dirties.context.problem.descriptor=Usage of @DirtiesContext makes integration tests slower\ninspection.dirties.context.use.quickfix=Removes usage of @DirtiesContextRegister bundle in plugin.xml:
<resource-bundle>com.auto1.intellij.tutorial.PluginBundle</resource-bundle>And finally use bundle in inspection, for example:
\n public String getDisplayName() {\n return PluginBundle.message(\"inspection.dirties.context.display.name\");\n }We picked LightPlatformCodeInsightFixtureTestCase as base for our tests because it is recommended in the documentation. Unfortunately, testing appeared harder to set up properly than expected.
First problem was that our tests couldn’t see classes from JDK, which was fixed by specifying project descriptor to use internal JDK:
\n protected LightProjectDescriptor getProjectDescriptor() {\n return new LightProjectDescriptor() {\n public Sdk getSdk() {\n return JavaAwareProjectJdkTableImpl.getInstanceEx().getInternalJdk();\n }\n };\n }Second problem was that visitor received incorrect fully qualified class name of the annotation. Instead of org.springframework.test.annotation.DirtiesContext, it got DirtiesContext while it worked fine for real project in IDE. It turns out that such behaviour occurs when test project doesn’t see definition of some class. This is fixable by either hardcoding problematic class into test or by adding dependency as library to the project. Second approach avoids copying source code from other projects and seems to be more interesting so it will be presented here. In order to download dependency jar we use ShrinkWrap library:
private File[] getMavenArtifacts(String... mavenArtifacts) {\n File[] files = Maven.resolver()\n .resolve(mavenArtifacts)\n .withoutTransitivity()\n .asFile();\n if (files.length == 0) {\n throw new IllegalArgumentException(\"Failed to resolve artifacts \" + Arrays.toString(mavenArtifacts));\n }\n return files;\n }When dependency is resolved and downloaded into local Maven cache it can be added as library with code listed below:
\n protected void attachMavenLibrary(String mavenArtifact) {\n File[] jars = getMavenArtifacts(mavenArtifact);\n Arrays.stream(jars).forEach(jar -> {\n String name = jar.getName();\n PsiTestUtil.addLibrary(myFixture.getProjectDisposable(), myModule, name, jar.getParent(), name);\n });\n }It’s important to use myFixture.getProjectDisposable() instead of myFixture.getProject() otherwise there is an exception during test shutdown:
com.intellij.openapi.util.TraceableDisposable$DisposalException: Virtual pointer 'jar:///somePath/.m2/repository/org/springframework/spring-test/5.1.5.RELEASE/spring-test-5.1.5.RELEASE.jar!/' hasn't been disposedNext surprise is that by default test searches for test data in strange location inside IntelliJ home folder which can be fixed with overriding getTestDataPath. Since input files most likely won’t compile because of missing imports and possible usage of special markers like src/test/java folder to store them:
@Override\n protected String getTestDataPath() {\n return \"src/test/testData\";\n }Inspection tests can be done by providing file to analyse and resulting file that should be created after given quick fix has been applied. Inspection tests use configureByFile to load input file, doHighlighting to trigger source code analysis, launchAction to execute quick fix and finally checkResultByFile to compare results against after file.
myFixture.configureByFile(testName + \".java\");\n myFixture.enableInspections(new DirtiesContextInspection());\n myFixture.doHighlighting();\n IntentionAction quickFixAction = myFixture.findSingleIntention(intentionHint);\n myFixture.launchAction(quickFixAction);\n myFixture.checkResultByFile(testName + \".after.java\");Complete source code can be found at github github
\nIf you would like to share your plugin with other developers, you can publish to JetBrains plugin repository as described in the documentation. Other simple option is to execute buildPlugin Gradle task which will create plugin zip file inside build/distributions and then install it via Install Plugin from disk... available inside Preferences > Plugins (in IntelliJ 2018.3 is available through \"gears icon\").
\nWhen working on your own ideas you might run into a situation when you don't know how to implement some functionality. In this situation you could try to find the answer using links provided in section below. What also worked for us was reading source code of inspections available as part of community edition of IntelliJ, often there is an existing inspection which does a similar thing to what you might want to do.
\nRESTful API’s are ruling the industry nowadays. They are very often associated with an extremely popular data format - JSON.\nIt is so popular that database creators decided to implement it directly in the database engines.\nAll modern databases offer JSON processing capabilities.\nLet’s have a look and see how can we make use of them to improve our application’s performance.
\nThe typical choice for a Java application is to use JPA as the persistence layer abstraction.\nIt became the de-facto standard for a reason - it offers great flexibility and ease of use.\nUnfortunately it doesn't come without a price since we have to pay with degraded performance.
\nIt is a widely known fact that ORM’s impact performance, compared to pure JDBC.\nOn the other hand, relatively few are willing to use plain JDBC, considering the amount of additional work it requires.
\nLet me demonstrate how to use JDBC to leverage the performance, with the help of modern Postgres features.
\nI have prepared a demo application running on Spring Boot which exposes example data from the northwind database.\nThe data are exposed in three different ways.
\nThe baseline version simply exposes JPA entities:
\n@GetMapping(path = \"/v1/orders/{order_id}\",\n produces = APPLICATION_JSON_VALUE)\npublic Order getEntity(@PathVariable(\"order_id\") Integer orderId) {\n return orderRepository.findById(orderId).orElseThrow();\n}The Order entity aggregates data from numerous database tables: orders, order_details, products, customers, customer_customer_demo and customer_demographics.\nYou could check the corresponding JPA mapping here.
Thanks to the magic of Spring the entity is being translated into a JSON response:
\n{\n \"orderId\": 10284,\n \"employeeId\": 4,\n \"orderDate\": \"1996-08-19\",\n \"requiredDate\": \"1996-09-16\",\n \"shippedDate\": \"1996-08-27\",\n \"shipVia\": 1,\n \"freight\": 76.56,\n \"shipName\": \"Lehmanns Marktstand\",\n \"shipAddress\": \"Magazinweg 7\",\n \"shipCity\": \"Frankfurt a.M.\",\n \"shipRegion\": null,\n \"shipPostalCode\": \"60528\",\n \"shipCountry\": \"Germany\",\n \"orderDetails\": [\n {\n \"orderId\": 10284,\n \"productId\": 27,\n \"product\": {\n \"productId\": 27,\n \"categoryId\": 3,\n \"supplierId\": 11,\n \"productName\": \"Schoggi Schokolade\",\n \"discontinued\": false,\n \"quantityPerUnit\": \"100 - 100 g pieces\",\n \"unitPrice\": 43.9,\n \"unitsInStock\": 49,\n \"unitsOnOrder\": 0,\n \"reorderLevel\": 30\n },\n \"unitPrice\": 35.1,\n \"quantity\": 15,\n \"discount\": 0.25\n },\n {...},\n {...},\n {...}\n ],\n \"customer\": {\n \"customerId\": \"LEHMS\",\n \"companyName\": \"Lehmanns Marktstand\",\n \"contactName\": \"Renate Messner\",\n \"contactTitle\": \"Sales Representative\",\n \"address\": \"Magazinweg 7\",\n \"city\": \"Frankfurt a.M.\",\n \"region\": null,\n \"postalCode\": \"60528\",\n \"country\": \"Germany\",\n \"phone\": \"069-0245984\",\n \"fax\": \"069-0245874\",\n \"customerDesc\": [\n {\n \"customerTypeId\": \"AAAA\",\n \"customerDesc\": \"First demographic\"\n },\n {...},\n {...},\n {...}\n ]\n }\n}The demo project includes the p6spy library which gives an insight into the queries being issued against database.\nTurns out that three queries are required to load a particular Order - this means three database round trips.
\nOf course we can force the ORM to join all the tables and fetch them in one hop, but this seems a not so elegant solution -\nit pushes the ORM to load duplicated data and the necessity of deduplicating them in-memory. Can we do better?
\nJDBC is always a choice if the solution could be expressed via SQL query. Luckily we can create one which matches our case perfectly:
\nWITH products(\"productId\", \"categoryId\", \"supplierId\", \"productName\", \"discontinued\", \"quantityPerUnit\",\n \"unitPrice\", \"unitsInStock\", \"unitsOnOrder\", \"reorderLevel\") AS\n (SELECT p.product_id,\n p.category_id,\n p.supplier_id,\n p.product_name,\n p.discontinued::boolean,\n p.quantity_per_unit,\n p.unit_price,\n p.units_in_stock,\n p.units_on_order,\n p.reorder_level\n FROM products p),\n orderDetails(\"orderId\", \"productId\", \"product\", \"unitPrice\", \"quantity\", \"discount\") AS\n (SELECT d.order_id,\n d.product_id,\n (SELECT to_json(p) FROM products p WHERE p.\"productId\" = d.product_id),\n d.unit_price,\n d.quantity,\n d.discount\n FROM order_details d),\n\n demographics(\"customerId\", \"customerTypeId\", \"customerDesc\") AS\n (SELECT ccd.customer_id,\n ccd.customer_type_id,\n cd.customer_desc\n FROM customer_demographics cd\n JOIN customer_customer_demo ccd on cd.customer_type_id = ccd.customer_type_id),\n customers(\"customerId\", \"companyName\", \"contactName\", \"contactTitle\", \"address\", \"city\", \"region\", \"postalCode\",\n \"country\", \"phone\", \"fax\", \"customerDesc\") AS\n (SELECT c.customer_id,\n c.company_name,\n c.contact_name,\n c.contact_title,\n c.address,\n c.city,\n c.region,\n c.postal_code,\n c.country,\n c.phone,\n c.fax,\n (SELECT to_json(array_agg(to_json(d))) FROM (SELECT \"customerTypeId\", \"customerDesc\" FROM demographics WHERE demographics.\"customerId\" = c.customer_id) d)\n FROM customers c),\n\n orders(\"orderId\", \"employeeId\", \"orderDate\", \"requiredDate\", \"shippedDate\", \"shipVia\", \"freight\",\n \"shipName\", \"shipAddress\", \"shipCity\", \"shipRegion\", \"shipPostalCode\", \"shipCountry\", \"orderDetails\",\n \"customer\") AS\n (SELECT o.order_id,\n o.employee_id,\n o.order_date,\n o.required_date,\n o.shipped_date,\n o.ship_via,\n o.freight,\n o.ship_name,\n o.ship_address,\n o.ship_city,\n o.ship_region,\n o.ship_postal_code,\n o.ship_country,\n (SELECT to_json(array_agg(to_json(d))) FROM orderDetails d WHERE d.\"orderId\" = o.order_id),\n (SELECT to_json(c) FROM customers c WHERE c.\"customerId\" = o.customer_id)\n FROM orders o)\n\nSELECT to_json(orders) AS json_body FROM orders WHERE orders.\"orderId\" = ?The query looks verbose, perhaps a little enigmatic.\nIt is a bit long and therefore it could be overwhelming.\nHowever, on a closer look we could notice it isn't so much complicated - there are few idioms which make the query powerful.
\nThe good news is that the query generates the exact same JSON as the JPA endpoint provides - it is possible thanks to Postgres’ excellent support for JSON format.\nOur case is not sophisticated though - single to_json function is enough to deal with JSON objects and arrays.
One can notice how the nesting has been realized: how to nest JSON objects and JSON arrays, e.g. the orderDetails or the customer into order).\nThe simple idiom (accordingly to_json(...) or to_json(array_agg(to_json(...)))) does the job.
The database query could be adapted according to the caller needs - the response can be shrinked, making the query simpler and - most probably - even more effective.
\nWorth noticing is that the join table customer_customer_demo has been mitigated in the end result, just like the JPA version does by using @ManyToMany.
\n\nThe query is flexible enough to substitute all JPA features we need.
\n
I want also emphasize the fact of CTE’s being used to decompose the query into logical parts.\nPersonally I find it elegant because it is easy on the eyes and it “scales” - meaning while our case gets more complicated (JSON aggregates more data, becomes deeply nested) our query gets longer instead of getting deeper.\nIf the query grows too long we can simplify it by extracting some it’s parts as database views.
\nThe CTE has one more advantage here: it gives a control over the keys visible in end resulting JSON (order_id becomes orderId).
So we have a nice query which allows us to fetch required data in one hop in a format ready to be consumed by our clients.\nThis means no adjustments are required, therefore the Java implementation can be that easy:
\n@SneakyThrows\n@GetMapping(path = \"/v2/orders/{order_id}\",\n produces = APPLICATION_JSON_VALUE)\npublic String getRawBodyFromDatabase(\n @PathVariable(\"order_id\") Integer orderId) {\n return jdbcTemplate.query(JSON_QUERY, (ResultSet rs) -> {\n rs.next();\n return rs.getString(1);\n }, orderId);\n}We didn’t use plain JDBC here but a JdbcTemplate which has a negligible overhead and it simplifies our code.
Of course we could achieve similar results using plain JDBC (e.g. by using PreparedStatement directly) - without the use of Spring.\nEven then the necessity of extracting only a single field from the result set makes the solution less coarse than using JDBC in a usual way\n(in our case it would require managing at least three result sets).\nAll the complexity has been moved down into the query.
Some folks could argue “I don’t want the JSON to be directly returned - I definitely need the POJO to make some processing over it”.\nThat's an absolutely fair statement! We can satisfy this requirement as well.
\nLet’s point it one more time: our query produces the exact same JSON as the Order JPA entity has been translated into.
\nThis means we can use the resulting JSON and unmarshall it into an object of class Order.\nOf course the object won’t be tracked by the JPA whatsoever but we don’t need these capabilities - remember: we are serving persisted data and not mutating them.
@SneakyThrows\n@GetMapping(path = \"/v3/orders/{order_id}\",\n produces = APPLICATION_JSON_VALUE)\npublic Order getEntityNotUsingJpa(@PathVariable(\"order_id\") Integer orderId) {\n return jdbcTemplate.query(JSON_QUERY, (ResultSet rs) -> {\n rs.next();\n return unmarshall(rs.getString(1), Order.class);\n }, orderId);\n}\n\n@SneakyThrows\nprivate <T> T unmarshall(String json_body, Class<T> clazz) {\n return objectMapper.readValue(json_body, clazz);\n}This version is very similar to V2 but it introduces an additional step of unmarshalling.\nIt is also a great verification step which makes us sure the V1 and V2 results are identical.
\nSo how are all the solutions performing?\nWe suspect the V2 version to be faster than the competitors.\nWe are also unsure about the V3 which requires additional unmarshalling phase - could it impact the performance badly?
\nThe demo project includes a JMeter configuration and a bash script which verifies the configurations.\nThe benchmark is always run against a fresh setup (db + app).\nThe app is running on java 11 with EpsilonGC turned on - this allows us to verify memory consumption of each solution.
\nThe requests are run in strictly defined sequence, fetching the orders one by one.\nThere’s a prewarming phase which issues 100 requests and then a profiling phase with 20k of requests.
\nI have run the benchmark several times on a Macbook Pro 2018, the results are similar on each run.\nHere’s a summary of the most recent run:
\nWe can see the V2 outperforming the V1 by 32 seconds - that’s 37% improvement.
\nWe can notice also a lower memory consumption on the V2 - it used 52 MB (49 MiB) less memory than V1\n(noted the initial app’s memory usage which is 785 MB):
\n\n\nWorth noticing: the V2 version does not leverage Hibernate at all therefore we could minimize the memory footprint even more by excluding the JPA dependencies.
\n
Let’s have a look at response times:
\nOnce again we are able to notice improved throughput as well as lower latency.
\nThe plot confirms that as well (cropped for brevity):
\nPersonally I am very well satisfied with the V3 results - the memory consumption being at the same level as the JPA it still offers satisfying performance, nearly as good as the V1.\nIt proves that the unmarshalling does not impact the performance as badly as we might expect.
\nI really encourage readers try running the benchmark and share their own results!
\nLast words about the database: the results are reproducible currently only on Postgres 12,\ndue to recent improvements in Common Table Expressions.
\nI have had a run on Postgres 11 and the V2’s performance was… horrible - I had to shorten the benchmark 10 times to make it finish in a reasonable time.\nHere are the results I got:
\nYou can give it a try - the demo app is prepared to allow a quick switch to Postgres 11.
\nThe query we have prepared enables fine control over the data being fetched from database.
\nWe are no longer restricted to tabular data - we could easily fetch complex data graphs, even manipulate them to match the desired format.\nWe did it keeping the solution maintainable and very effective in terms of throughput.
\nIt is still unknown to me what is the main source of latency in the JPA version: either is it the additional round trips or the internals of Hibernate.\nI see the topic interesting to explore.\nPerhaps some day we’ll get an ORM which is able to load entity graphs using a single query.\nAny Hibernate devs reading here?
","fields":{"slug":"/serving-json-from-postgres/","tags":["jdbc","jpa","postgres12","json","spring boot"]}}}]},"authorArticles":{"totalCount":1,"edges":[{"node":{"id":"24fd2709-272a-5811-a69c-7100924de21c","frontmatter":{"category":"Coding","title":"Serving JSON (almost) directly from database","date":"2019-10-29","summary":"Learn how to query JSON data from Postgres 12 to improve Java application's performance.","thumbnail":{"relativePath":"pages/serving-json-from-postgres/liberty.jpg","childImageSharp":{"resolutions":{"base64":"data:image/jpeg;base64,/9j/2wBDABALDA4MChAODQ4SERATGCgaGBYWGDEjJR0oOjM9PDkzODdASFxOQERXRTc4UG1RV19iZ2hnPk1xeXBkeFxlZ2P/2wBDARESEhgVGC8aGi9jQjhCY2NjY2NjY2NjY2NjY2NjY2NjY2NjY2NjY2NjY2NjY2NjY2NjY2NjY2NjY2NjY2NjY2P/wgARCAAUABQDASIAAhEBAxEB/8QAGAABAAMBAAAAAAAAAAAAAAAAAAIDBAX/xAAUAQEAAAAAAAAAAAAAAAAAAAAA/9oADAMBAAIQAxAAAAHryz0HUYxXEAP/xAAbEAABBAMAAAAAAAAAAAAAAAABAgMSIhAgIf/aAAgBAQABBQJw0SoKxLjVDPT/xAAUEQEAAAAAAAAAAAAAAAAAAAAg/9oACAEDAQE/AR//xAAUEQEAAAAAAAAAAAAAAAAAAAAg/9oACAECAQE/AR//xAAYEAACAwAAAAAAAAAAAAAAAAABIQAQIP/aAAgBAQAGPwJRGzn/xAAaEAEAAwADAAAAAAAAAAAAAAABABEhEDGB/9oACAEBAAE/IVEqq3yWWlcbozCOmOokIz//2gAMAwEAAgADAAAAEOAPAP/EABQRAQAAAAAAAAAAAAAAAAAAACD/2gAIAQMBAT8QH//EABQRAQAAAAAAAAAAAAAAAAAAACD/2gAIAQIBAT8QH//EABwQAQADAQADAQAAAAAAAAAAAAEAETEhQXGhwf/aAAgBAQABPxBIyYq1CL8uc5XU14YIGwTYJTDeFrX2G/k7OwjIAbIBek//2Q==","width":325,"height":325,"src":"/static/e0de7302edcea3d0b0c19a9d147b547a/a2998/liberty.jpg","srcSet":"/static/e0de7302edcea3d0b0c19a9d147b547a/a2998/liberty.jpg 1x,\n/static/e0de7302edcea3d0b0c19a9d147b547a/99cce/liberty.jpg 1.5x,\n/static/e0de7302edcea3d0b0c19a9d147b547a/6e995/liberty.jpg 2x,\n/static/e0de7302edcea3d0b0c19a9d147b547a/968b1/liberty.jpg 3x"}}},"authorName":"Mariusz Nowak","authorDescription":"Mariusz is a Senior Software Engineer at AUTO1 Group.","authorAvatar":null,"headerImage":null},"html":"RESTful API’s are ruling the industry nowadays. They are very often associated with an extremely popular data format - JSON.\nIt is so popular that database creators decided to implement it directly in the database engines.\nAll modern databases offer JSON processing capabilities.\nLet’s have a look and see how can we make use of them to improve our application’s performance.
\nThe typical choice for a Java application is to use JPA as the persistence layer abstraction.\nIt became the de-facto standard for a reason - it offers great flexibility and ease of use.\nUnfortunately it doesn't come without a price since we have to pay with degraded performance.
\nIt is a widely known fact that ORM’s impact performance, compared to pure JDBC.\nOn the other hand, relatively few are willing to use plain JDBC, considering the amount of additional work it requires.
\nLet me demonstrate how to use JDBC to leverage the performance, with the help of modern Postgres features.
\nI have prepared a demo application running on Spring Boot which exposes example data from the northwind database.\nThe data are exposed in three different ways.
\nThe baseline version simply exposes JPA entities:
\n@GetMapping(path = \"/v1/orders/{order_id}\",\n produces = APPLICATION_JSON_VALUE)\npublic Order getEntity(@PathVariable(\"order_id\") Integer orderId) {\n return orderRepository.findById(orderId).orElseThrow();\n}The Order entity aggregates data from numerous database tables: orders, order_details, products, customers, customer_customer_demo and customer_demographics.\nYou could check the corresponding JPA mapping here.
Thanks to the magic of Spring the entity is being translated into a JSON response:
\n{\n \"orderId\": 10284,\n \"employeeId\": 4,\n \"orderDate\": \"1996-08-19\",\n \"requiredDate\": \"1996-09-16\",\n \"shippedDate\": \"1996-08-27\",\n \"shipVia\": 1,\n \"freight\": 76.56,\n \"shipName\": \"Lehmanns Marktstand\",\n \"shipAddress\": \"Magazinweg 7\",\n \"shipCity\": \"Frankfurt a.M.\",\n \"shipRegion\": null,\n \"shipPostalCode\": \"60528\",\n \"shipCountry\": \"Germany\",\n \"orderDetails\": [\n {\n \"orderId\": 10284,\n \"productId\": 27,\n \"product\": {\n \"productId\": 27,\n \"categoryId\": 3,\n \"supplierId\": 11,\n \"productName\": \"Schoggi Schokolade\",\n \"discontinued\": false,\n \"quantityPerUnit\": \"100 - 100 g pieces\",\n \"unitPrice\": 43.9,\n \"unitsInStock\": 49,\n \"unitsOnOrder\": 0,\n \"reorderLevel\": 30\n },\n \"unitPrice\": 35.1,\n \"quantity\": 15,\n \"discount\": 0.25\n },\n {...},\n {...},\n {...}\n ],\n \"customer\": {\n \"customerId\": \"LEHMS\",\n \"companyName\": \"Lehmanns Marktstand\",\n \"contactName\": \"Renate Messner\",\n \"contactTitle\": \"Sales Representative\",\n \"address\": \"Magazinweg 7\",\n \"city\": \"Frankfurt a.M.\",\n \"region\": null,\n \"postalCode\": \"60528\",\n \"country\": \"Germany\",\n \"phone\": \"069-0245984\",\n \"fax\": \"069-0245874\",\n \"customerDesc\": [\n {\n \"customerTypeId\": \"AAAA\",\n \"customerDesc\": \"First demographic\"\n },\n {...},\n {...},\n {...}\n ]\n }\n}The demo project includes the p6spy library which gives an insight into the queries being issued against database.\nTurns out that three queries are required to load a particular Order - this means three database round trips.
\nOf course we can force the ORM to join all the tables and fetch them in one hop, but this seems a not so elegant solution -\nit pushes the ORM to load duplicated data and the necessity of deduplicating them in-memory. Can we do better?
\nJDBC is always a choice if the solution could be expressed via SQL query. Luckily we can create one which matches our case perfectly:
\nWITH products(\"productId\", \"categoryId\", \"supplierId\", \"productName\", \"discontinued\", \"quantityPerUnit\",\n \"unitPrice\", \"unitsInStock\", \"unitsOnOrder\", \"reorderLevel\") AS\n (SELECT p.product_id,\n p.category_id,\n p.supplier_id,\n p.product_name,\n p.discontinued::boolean,\n p.quantity_per_unit,\n p.unit_price,\n p.units_in_stock,\n p.units_on_order,\n p.reorder_level\n FROM products p),\n orderDetails(\"orderId\", \"productId\", \"product\", \"unitPrice\", \"quantity\", \"discount\") AS\n (SELECT d.order_id,\n d.product_id,\n (SELECT to_json(p) FROM products p WHERE p.\"productId\" = d.product_id),\n d.unit_price,\n d.quantity,\n d.discount\n FROM order_details d),\n\n demographics(\"customerId\", \"customerTypeId\", \"customerDesc\") AS\n (SELECT ccd.customer_id,\n ccd.customer_type_id,\n cd.customer_desc\n FROM customer_demographics cd\n JOIN customer_customer_demo ccd on cd.customer_type_id = ccd.customer_type_id),\n customers(\"customerId\", \"companyName\", \"contactName\", \"contactTitle\", \"address\", \"city\", \"region\", \"postalCode\",\n \"country\", \"phone\", \"fax\", \"customerDesc\") AS\n (SELECT c.customer_id,\n c.company_name,\n c.contact_name,\n c.contact_title,\n c.address,\n c.city,\n c.region,\n c.postal_code,\n c.country,\n c.phone,\n c.fax,\n (SELECT to_json(array_agg(to_json(d))) FROM (SELECT \"customerTypeId\", \"customerDesc\" FROM demographics WHERE demographics.\"customerId\" = c.customer_id) d)\n FROM customers c),\n\n orders(\"orderId\", \"employeeId\", \"orderDate\", \"requiredDate\", \"shippedDate\", \"shipVia\", \"freight\",\n \"shipName\", \"shipAddress\", \"shipCity\", \"shipRegion\", \"shipPostalCode\", \"shipCountry\", \"orderDetails\",\n \"customer\") AS\n (SELECT o.order_id,\n o.employee_id,\n o.order_date,\n o.required_date,\n o.shipped_date,\n o.ship_via,\n o.freight,\n o.ship_name,\n o.ship_address,\n o.ship_city,\n o.ship_region,\n o.ship_postal_code,\n o.ship_country,\n (SELECT to_json(array_agg(to_json(d))) FROM orderDetails d WHERE d.\"orderId\" = o.order_id),\n (SELECT to_json(c) FROM customers c WHERE c.\"customerId\" = o.customer_id)\n FROM orders o)\n\nSELECT to_json(orders) AS json_body FROM orders WHERE orders.\"orderId\" = ?The query looks verbose, perhaps a little enigmatic.\nIt is a bit long and therefore it could be overwhelming.\nHowever, on a closer look we could notice it isn't so much complicated - there are few idioms which make the query powerful.
\nThe good news is that the query generates the exact same JSON as the JPA endpoint provides - it is possible thanks to Postgres’ excellent support for JSON format.\nOur case is not sophisticated though - single to_json function is enough to deal with JSON objects and arrays.
One can notice how the nesting has been realized: how to nest JSON objects and JSON arrays, e.g. the orderDetails or the customer into order).\nThe simple idiom (accordingly to_json(...) or to_json(array_agg(to_json(...)))) does the job.
The database query could be adapted according to the caller needs - the response can be shrinked, making the query simpler and - most probably - even more effective.
\nWorth noticing is that the join table customer_customer_demo has been mitigated in the end result, just like the JPA version does by using @ManyToMany.
\n\nThe query is flexible enough to substitute all JPA features we need.
\n
I want also emphasize the fact of CTE’s being used to decompose the query into logical parts.\nPersonally I find it elegant because it is easy on the eyes and it “scales” - meaning while our case gets more complicated (JSON aggregates more data, becomes deeply nested) our query gets longer instead of getting deeper.\nIf the query grows too long we can simplify it by extracting some it’s parts as database views.
\nThe CTE has one more advantage here: it gives a control over the keys visible in end resulting JSON (order_id becomes orderId).
So we have a nice query which allows us to fetch required data in one hop in a format ready to be consumed by our clients.\nThis means no adjustments are required, therefore the Java implementation can be that easy:
\n@SneakyThrows\n@GetMapping(path = \"/v2/orders/{order_id}\",\n produces = APPLICATION_JSON_VALUE)\npublic String getRawBodyFromDatabase(\n @PathVariable(\"order_id\") Integer orderId) {\n return jdbcTemplate.query(JSON_QUERY, (ResultSet rs) -> {\n rs.next();\n return rs.getString(1);\n }, orderId);\n}We didn’t use plain JDBC here but a JdbcTemplate which has a negligible overhead and it simplifies our code.
Of course we could achieve similar results using plain JDBC (e.g. by using PreparedStatement directly) - without the use of Spring.\nEven then the necessity of extracting only a single field from the result set makes the solution less coarse than using JDBC in a usual way\n(in our case it would require managing at least three result sets).\nAll the complexity has been moved down into the query.
Some folks could argue “I don’t want the JSON to be directly returned - I definitely need the POJO to make some processing over it”.\nThat's an absolutely fair statement! We can satisfy this requirement as well.
\nLet’s point it one more time: our query produces the exact same JSON as the Order JPA entity has been translated into.
\nThis means we can use the resulting JSON and unmarshall it into an object of class Order.\nOf course the object won’t be tracked by the JPA whatsoever but we don’t need these capabilities - remember: we are serving persisted data and not mutating them.
@SneakyThrows\n@GetMapping(path = \"/v3/orders/{order_id}\",\n produces = APPLICATION_JSON_VALUE)\npublic Order getEntityNotUsingJpa(@PathVariable(\"order_id\") Integer orderId) {\n return jdbcTemplate.query(JSON_QUERY, (ResultSet rs) -> {\n rs.next();\n return unmarshall(rs.getString(1), Order.class);\n }, orderId);\n}\n\n@SneakyThrows\nprivate <T> T unmarshall(String json_body, Class<T> clazz) {\n return objectMapper.readValue(json_body, clazz);\n}This version is very similar to V2 but it introduces an additional step of unmarshalling.\nIt is also a great verification step which makes us sure the V1 and V2 results are identical.
\nSo how are all the solutions performing?\nWe suspect the V2 version to be faster than the competitors.\nWe are also unsure about the V3 which requires additional unmarshalling phase - could it impact the performance badly?
\nThe demo project includes a JMeter configuration and a bash script which verifies the configurations.\nThe benchmark is always run against a fresh setup (db + app).\nThe app is running on java 11 with EpsilonGC turned on - this allows us to verify memory consumption of each solution.
\nThe requests are run in strictly defined sequence, fetching the orders one by one.\nThere’s a prewarming phase which issues 100 requests and then a profiling phase with 20k of requests.
\nI have run the benchmark several times on a Macbook Pro 2018, the results are similar on each run.\nHere’s a summary of the most recent run:
\nWe can see the V2 outperforming the V1 by 32 seconds - that’s 37% improvement.
\nWe can notice also a lower memory consumption on the V2 - it used 52 MB (49 MiB) less memory than V1\n(noted the initial app’s memory usage which is 785 MB):
\n\n\nWorth noticing: the V2 version does not leverage Hibernate at all therefore we could minimize the memory footprint even more by excluding the JPA dependencies.
\n
Let’s have a look at response times:
\nOnce again we are able to notice improved throughput as well as lower latency.
\nThe plot confirms that as well (cropped for brevity):
\nPersonally I am very well satisfied with the V3 results - the memory consumption being at the same level as the JPA it still offers satisfying performance, nearly as good as the V1.\nIt proves that the unmarshalling does not impact the performance as badly as we might expect.
\nI really encourage readers try running the benchmark and share their own results!
\nLast words about the database: the results are reproducible currently only on Postgres 12,\ndue to recent improvements in Common Table Expressions.
\nI have had a run on Postgres 11 and the V2’s performance was… horrible - I had to shorten the benchmark 10 times to make it finish in a reasonable time.\nHere are the results I got:
\nYou can give it a try - the demo app is prepared to allow a quick switch to Postgres 11.
\nThe query we have prepared enables fine control over the data being fetched from database.
\nWe are no longer restricted to tabular data - we could easily fetch complex data graphs, even manipulate them to match the desired format.\nWe did it keeping the solution maintainable and very effective in terms of throughput.
\nIt is still unknown to me what is the main source of latency in the JPA version: either is it the additional round trips or the internals of Hibernate.\nI see the topic interesting to explore.\nPerhaps some day we’ll get an ORM which is able to load entity graphs using a single query.\nAny Hibernate devs reading here?
","fields":{"slug":"/serving-json-from-postgres/","tags":["jdbc","jpa","postgres12","json","spring boot"]}}}]}},"pageContext":{"slug":"/postgres12-a-precious-release/","tags":["postgres","postgres12","release","sql","cte"],"category":"Coding","author":"Mariusz Nowak","date":"2019-10-29"}}