Annotating images and taking notes in APEX

One of the things we love about web development these days is that pretty much anything is possible. If you follow enough blogs, tweets, social media posts you’re bound to be inspired. You will also know what’s possible when you face a particular requirement, which others may not know, or deem impossible.

I’m pretty sure there’s a bunch of applications out there that were NOT built in APEX because there was a requirement for something that APEX could NOT do out of the box. This is something that we try to solve. We want you to know that whatever can be done by some other application can be done in APEX, and you can make it super easy to reuse by turning it into an APEX plug-in.

We recently found ourselves in this exact situation being tasked with building a plug-in for a complex interactive component in APEX for the Queensland Government in Australia. We accepted this challenge without knowing whether it would be possible or how long it would take. Pretty risky right? Let me explain….

The Requirement

Back in May 2020 we had a enquiry from the department of Queensland Health to build a custom solution to assist in breast cancer screening. They are in the process of migrating their existing complex “Oracle Forms” breast cancer screening application to APEX.

One key part of the existing Forms application is that the clinicians annotate an image during a patient screening. Marking the image with things like scarring, redness, bruising, lumps etc. and (optionally) record some notes about each particular marker during the patients visit. These can then be read back and added to the “canvas” on the patients next visit to the hospital i.e. so the markers, their position, and associated comments must be stored separately from the actual image.

What we needed to replace

Here’s a screenshot of a Java component from their existing system to give you an idea of what we needed to replace. It’s not the prettiest of components and could benefit from some modernization…

At the time of this request we had recently developed a file upload and image editor plugin which caught the attention of the lead developer on this project, since you could edit & annotate images and save the merged result. His line of thinking was that he had an image that needed annotation, so surely we can just make a few changes and we are good to go, right? Is it ever that easy?

The Image editor plugin that we built integrates an existing Image Editor web component called Doka, developed by a company called PQINA in the Netherlands. This is what the latest version of Doka looks like on their demo site:

Looking at the above image do you think it would be an obvious candidate to replace the Java component? I recommend that you visit the site and check out the demo and see what you think after a bit more investigation, but first you need to be aware of their specific requirements….

The Specifics

In their requirements document we had to provide the following capabilities:

• the ability to select individual markings and record some text comments
• if a comment is made then the marker needs to show a “C” in the top left corner
• the ability to resize the markers
• the ability to rotate the markers
• the ability to change the background image e.g. to show left/right mastectomy
•  the ability to predefine markers e.g. for breast implants
• the ability to disable certain markers from being used e.g. based on access levels
• the ability to prevent certain markers from being modified or removed
• the ability to switch between editing and read only mode

Did you check out the Doka demo? Would you feel comfortable that it could satisfy all the above requirements?

Based on our previous experience some of these things sounded a bit daunting to provide, especially given the fact that we did not know of any other product out there that would provide all this, but when you’re a small business you find yourself saying “Yes” more times than you say “No”. Personally I love something challenging like this.

Previous Experience Helps (a lot)

I must point out that I was in a similar position 10 years ago when asked to create an interactive Body Chart for a physiotherapy company when I was a freelancer. Having successfully achieved that requirement within 4 days, it gave me a huge boost of confidence and was a real turning point in my career. Here’s a (bad quality) video of the plugin I built for APEX 4.x using Raphael.js. It also needed to support IE7 back then, thankfully today everything is much easier 🙂

 

 

Research & Development

We initially contacted the vendor in May 2020 trying to devise a possible strategy how we could use their existing component. This annotated image was our original request for missing features that we thought we would need. I thought it was kind of funny that we annotated an image, of an image annotator.

Surprisingly the vendor responded saying that he was already working on a new marker/sticker feature, but it was not yet available. He provided us with a beta version and we performed a pretty thorough evaluation. We found that using the new marker/sticker functionality we were able to tick off several requirements immediately e.g. adding markers, resizing markers, removing markers. We also had a look at the updated documentation and associated API about what additional possibilities. There was enough to convince us to use Doka for this requirement.

Vendor Support

One thing we do when building a plugin in APEX based on a 3rd party component is that we reach out to the vendor (in this case PQINA – Rik Schennink) and establish an OEM license agreement and direct support. Since we already had this agreement in place for our “File Upload & Image Editor” it was something we didn’t need to do. We had previously established a good working relationship with Rik including a commercial support agreement. So we felt confident that we could overcome any blocking issue. As it turns out this was absolutely crucial and indeed a lifeline we had to use several times.

The Prototype

Since I was taking the lead on this project, I booked myself into a hotel for 4 days to work around the clock to produce the initial prototype, which would be a variation of our original Image Editor plugin. This meant I had a head start already. Personally I set myself a 4 day limit on whether something is achievable or not, it’s enough time to really get into everything, but also not too much time if you end up failing. Plus you usually have Friday left to dry your tears before the weekend comes.

These 4 days were basically a holiday, for everyone else it’s probably like forced detention. If you have children you’ll know why it’s a holiday. The possibility of getting this done whilst doing my regular day job and all the regular family duties would have probably resulted in a nervous breakdown. However 4 days of uninterrupted time doing something like this? It did the opposite, I loved every minute of it and it did my mental health wonders!

At the end of 4 days I was relieved to know that we’d be able to achieve what they needed. The prototype needed some refinement but the basics were in place and I knew the additional enhancements that we needed from the vendor.

Key Findings

We were provided GIF images for the background and markers by QLD Health. This didn’t look to good when resizing markers and resizing the background image on window resizing. The component supported SVG (Scalable Vector Graphics) image types that would scale nicely when resizing, so we recommended that they have a Graphic Designer create them.

For saving we needed to parse the annotation information provided in JSON format. Unlike days of old, this requirement is pretty easy these days thanks to the native JSON type in the database or worst case falling back to using APEX_JSON.

We also discovered that we needed 3 levels of dynamic refreshing (without reloading the page)

1) the background image

You could change the background image if the patient had a left/right/double mastectomy, or if they had breast implants.

2) the selectable markers

We needed to refresh the markers in some situations e.g. when they switch between enabled/disabled state. This can happen when either a marker limit might be reached i.e. you can only have 10 markers of each type added. Or it can happen when your security rules change.

3) the annotations

We would need to load the markers from the patients previous visit, if this was not their first visit. You could possibly switch between the annotation state across visits, like a history.

Requested Doka Enhancements

Whilst we able to achieve the majority of the requirements with the existing image editor, we did have to request the following enhancements that ate into our budget:

• An API method to select a marker using it’s ID value
• Provide a series of event hooks for things like “Before Marker Selection”, “After Marker Selection”, “Add Markup”, “Delete Markup”, “Update Markup” so we could perform targeted actions
• An ability to disable a marker from being added to the canvas i.e. based security access levels
• Show a tooltip on hover when selecting a marker to add to the canvas
• Prevent the next marker being selected when you delete an existing marker and hiding the delete button
• Prevent a marker from being deselected when you select outside of the canvas e.g. when entering comments
• Change the resize handles hit area size so markers can be resized very small and still allow them to be dragged
• The option to hide the top level tabs e.g. crop, filter, colors, markup

The point here is that fixed price projects have their drawbacks, but being able to finish the project and provide the full functionality requested is what’s most important. It does take quite some time and back and forth to get things right. Make sure you do your home work on initial pricing and include some buffer for similar situations.

The Final Result

We were able to provide all the required functionality except for marker rotation, which thankfully has been added to the latest version of Doka which we are in the process of upgrading to.

We were even able to integrate the component with the interactive grid. If you watch the demo video below you can see how we can automatically select the associated grid row when selecting a marker and conversely select the marker automatically when selecting the associated grid row. We can thank event hooks on both Doka and the Interactive grid that enable us to achieve this 2 way binding.

 

Link

Embed

Copy and paste this HTML code into your webpage to embed.

Commercial Plugin

We are currently working on the upgrade to the latest version of Doka and creating demos and documentation so that anyone could license and use this component in their APEX application(s). You will be able to customize the following:

• You can choose any background image you like and switch them dynamically
• You provide any set of markers you like, we recommend using SVG images for scalability reasons when resizing
• You can use on mobile or desktop devices
• You can bind multiple page items to a marker to record information, allowing the user to perform all their annotations before saving
• You have the option to auto save changes
• You can enable additional pre-defined markup features like arrows, circles, rectangles, text etc. and even use all the available image editing functionality if it’s needed
• You can optionally save a merged image in addition to the normalizing the annotation JSON to tables/tapi etc.

Whilst this component was designed for breast cancer screening, it could be used for things like:

• annotating X-rays, body charts etc.
• the Insurance claim industry where you can annotate images you are submitting to show damage on cars, shipped goods etc.
• Car/House rentals where you can mark the damage and record notes about the particular dents/scratches etc.
• Real estate websites, you could annotate features of the kitchen, bathroom, bedrooms and make it interactive for people viewing the pictures online
• Annotating schematic drawings e.g. architecture plans, blue prints etc.
• Making custom diagrams, your markers could be a set of shapes and you can connect them using arrows, text and shapes
• Health and Safety, you can take an uploaded picture of any issue in the workplace, and make detailed notes and annotations highlighting problem areas that need attention
• Reporting application issues i.e. you can integrate screen capture in your app and allow the user to take a screenshot of the current issue, annotate the problem areas with associated notes
• Can you think of something?

Live Collaboration

It is also possible for us to take this to the next level by providing websocket support allowing multiple people to live annotate an image together realtime. So if you stumble across this post looking for an annotation solution with this  capability then we can build it in to the plugin if you are willing to contribute to the development cost. We have previously built live editing into our enterprise framework which you can see here for an idea of how this could look/work.

In Summary

The goal of this post is to highlight the great integration capabilities of APEX and how you can get existing APEX components like the page items & the Interactive Grid working in tandem with 3rd party web components.

Whilst annotating images might seem like a niche requirement, it might have more use cases than you think. There is a lot of power storing image annotations separately from the image. You can turn a static image into an interactive one, with a fine grained set of functionality that is versatile for different use cases.

They say a picture speaks a thousand words, well in the case you can actually write those thousand words for every annotation you want to add to your image. Only if you use APEX of course 😉

 

P.S. Feel free to leave a comment, we’d like to hear your feedback, or if you have any ideas on what you think this solution might also be useful for?