OraTransplant

Patch 9.0.4.2.0 misses MouseWheelHandler.class

We’ve seen a dramatic drop in performance of Oracle Forms since we installed patch set 9.0.4.2.0. As it turns out this is caused by a missing class called MouseWheelHandler. In this blog entry I’ll describe how I found out about it and how we fixed it. Looking at the Java Console (right-mouse click on the Java icon in your system tray and select Show Console) we could see hundreds of the same exceptions:

sun.plugin.cache.DownloadException
	at sun.plugin.cache.CachedFileLoader.load(Unknown Source)
	at sun.plugin.cache.FileCache.get(Unknown Source)
	at sun.net.www.protocol.https.PluginDelegateHttpsURLConnection.connectWithCache(Unknown Source)
	at sun.net.www.protocol.https.PluginDelegateHttpsURLConnection.connect(Unknown Source)
	at sun.net.www.protocol.https.PluginDelegateHttpsURLConnection.getInputStream(Unknown Source)
	at java.net.HttpURLConnection.getResponseCode(Unknown Source)
	at sun.net.www.protocol.https.HttpsURLConnectionImpl.getResponseCode(Unknown Source)
	at sun.applet.AppletClassLoader.getBytes(Unknown Source)
	at sun.applet.AppletClassLoader.access$100(Unknown Source)
	at sun.applet.AppletClassLoader$1.run(Unknown Source)
	at java.security.AccessController.doPrivileged(Native Method)
	at sun.applet.AppletClassLoader.findClass(Unknown Source)
	at java.lang.ClassLoader.loadClass(Unknown Source)
	at sun.applet.AppletClassLoader.loadClass(Unknown Source)
	at java.lang.ClassLoader.loadClass(Unknown Source)
	at java.lang.ClassLoader.loadClassInternal(Unknown Source)
	at java.lang.Class.forName0(Native Method)
	at java.lang.Class.forName(Unknown Source)
	at oracle.ewt.scrolling.MouseWheelUtils._getMouseWheelHandlerClass(Unknown Source)
	at oracle.ewt.scrolling.MouseWheelUtils._getMouseWheelHandlerInstance(Unknown Source)
	at oracle.ewt.scrolling.MouseWheelUtils.addMouseWheelListener(Unknown Source)
	at oracle.ewt.scrolling.scrollBox.ScrollBox.setWheelScrollingEnabled(Unknown Source)
	at oracle.ewt.scrolling.scrollBox.ScrollBox.(Unknown Source)
	at oracle.forms.ui.FScrollBox.(Unknown Source)
	at oracle.forms.handler.FormCanvas.onCreate(Unknown Source)
	at oracle.forms.engine.Runform.onCreateHandler(Unknown Source)
	at oracle.forms.engine.Runform.processMessage(Unknown Source)
	at oracle.forms.engine.Runform.processSet(Unknown Source)
	at oracle.forms.engine.Runform.onMessageReal(Unknown Source)
	at oracle.forms.engine.Runform.onMessage(Unknown Source)
	at oracle.forms.engine.Runform.sendInitialMessage(Unknown Source)
	at oracle.forms.engine.Runform.startRunform(Unknown Source)
	at oracle.forms.engine.Main.createRunform(Unknown Source)
	at oracle.forms.engine.Main.start(Unknown Source)
	at sun.applet.AppletPanel.run(Unknown Source)
	at java.lang.Thread.run(Unknown Source)

Some more testing revealed that you get a bunch of these errors when opening a Form. It looks related to the number of items on the form. The more items you have, the more DownloadExceptions.

I’ve noticed before that Sun JPI raises such a DownloadException when retrieving a file that does not exist over SSL. That’s not really an issue since the file couldn’t be downloaded anyway. However, in this case it seems to cause quite some delay. It looks like handling the DownloadException costs quite some CPU resources. On low end machines (Celeron 600MHz) you can see the CPU load peek to 100% and stay there for a while. Opening on of our larger forms in Query mode (so without any database interaction) took 6 seconds before the 9.0.4.2.0 patch. After installing the patch we started getting these download exceptions and the startup time of this form increased to 28 seconds. That’s more than four times as long!

I then looked at the Apache logs to find out which file the Forms applet was trying to download. It turned out it is trying to download /forms90/java/oracle/ewt/scrolling/MouseWheelHandler.class over and over. You can also see this from the Java error stack. You can see a method oracle.ewt.scrolling.MouseWheelUtils._getMouseWheelHandlerClass. By the looks of its name it is probably to get a MouseWheelHandler class. That’s exactly the file it is trying to download.

Normally, the Forms applet doesn’t need to download additional class files during its operation. Everything it needs should be in the JAR file. I opened f90all.jar to have a look. I browsed to the directory oracle/ewt/scrolling as seen in the 404 errors in the Apache log. In that directory I found a MouseWheelScrolling.class and a MouseWheelUtils.class but no MouseWheelHandler.class.

I then took a copy of the f90all.jar for Forms 9.0.4.1.0 as we didn’t have the problems with that version. As it turns out none of these mouse wheel classes are in that version. It looks like Oracle should have added three classes to the 9.0.4.2.0 patch set, but forgot one.

It was at this moment that I logged a TAR at MetaLink to let Oracle have a look. The support engineer quickly found (unpublished) bug 4259148, which apparently is exactly about this issue. That bug was filed against version 10.x of Forms and was fixed in the final release of 10.1.2.0.2. At some stage the MouseWheelHandler.class was also missing from an (internal) version 10.x of Forms and it was fixed before the final release.

This gave me the idea to download the just release 10.1.2.0.2 version and have a look at the JAR file. All three mouse wheel classes are there! I checked the signature of the two that were already in the 9.0.4.2.0 version and they’re exactly the same. If they haven’t changed since 9.0.4.2.0, then there might be a good change the extra MouseWheelHandler class might just work in version 9.0.4.2.0.

During all this investigation of my own, the Oracle support engineer (Roelof) kept pushing Development for a fix, since it was a major hit on performance for our system and complaints were very frequent. I must say that Roelof did a great job. He was right on top of it and really helped me a lot. After assuring Oracle that we couldn’t upgrade to 10.1.2.0.2 at this moment, they agreed to make a fix for the issue in version 9.0.4.2.0.

But we really needed the fix ASAP, since we were going to shutdown our old Citrix client-server Forms 6i environment. We have been running this in parallel with the new Forms 10g systems for two months. We really needed to shutdown the Citrix systems but we users were just refusing to use the Forms 10g systems because of the performance.

In the end, I took the risk and repackaged the JAR file myself adding the missing class file from the v10.1.2.0.2 JAR file. This even gave me the opportunity to re-sign the JAR file myself, getting rid of the confusion certificate by “Developer/Oracle”. It’s all documented at OTN and on my blog.

This completely fixed the problem and we’re back at the performance we have before the 9.0.4.2.0 patch. Oracle is still working on the official patch, but I guess (hope) that they are doing exactly the same as I did: just adding the MouseWheelHandler.class from Forms 10.1.2.0.2 to the f90all.jar of Forms 9.0.4.2.0.

So, if you are having performance problems of your own since the upgrade of Forms 9.0.4.2.0 this might be one of the reasons.

Other difficult to diagnose issues we had impacting performance are a faulty DNS server and having large amounts of data on your Windows clipboard.

Update 19-sep-2005: Oracle has just release a one-off patch for the issue. I didn’t have a chance to look at it yet, since I’m at Oracle OpenWorld in San Francisco. If you want to have a look for yourself just find patch 4259148. Looking at the ReadMe it’s just the new JAR files with (I guess) the missing MouseWheel class.

Function key not allowed bug?

Ever since we migrated from Forms 6i to Forms 10g (9.0.4) we had user complaining about getting sporadic errors “FRM-41009 Function key not allowed. Press Ctrl+F1 for list of valid keys”. Once they’re in this situation every key pressed will raise this error. The only key that Forms responds to is Ctrl-Q to close the Form. When the form is closed, the application functions normally and all keys can be used again.

I already logged a TAR a couple of weeks ago but without a reproducable testcase there wasn’t much support could do for us. I was hoping they could find some known bug, but they couldn’t. The only thing we could do is inform our users of the issue and asking them if they could come up with the steps to reproduce.

Today we got a break. One of our users called she could reproduce the problem at will. I joined here and she was right. I wrote down the (quite a few) steps to reproduce and tried it at my own test-setup. Fortunately the problem also reproduced there.

I then figured out which forms, menus and libraries were needed to reproduce the case. It turned out I need 26 forms, libraries and menus for the smallest testcase. I doubt Oracle Support would be happy with such a testcase

I also did some searching on Google and Metalink and could find other users with the same problems, but none with a fix. I’ll try to trim down the testcase to a reasonable size in the coming days and will then open a TAR and have Support look at it. Let’s hope they can come up with a fix. I’ll update this blog entry when I have a solution.

Update 30-aug-2005: As it turns out, the error is caused by juggling around with the menus. First to give some more information on the testcase I provided to Oracle:

1. User starts the application with a form that has the menu property set.
2. User clicks a menu entry that does a CALL_FORM('form1', hide, do_replace);
   This form has the menu property set and so the menu is replaced. In our case this was the same menu so you cannot actually see this.
3. User clicks a menu entry that does a NEW_FORM('form2');
   This form does not have the menu property set
4. User clicks a menu entry that does a NEW_FORM('form1');
   This call causes the “FRM-41009 Function key not allowed”

Oracle and I found three workarounds:

1. Setting the menu property of form2 to the same menu
2. Use OPEN_FORM in stead of NEW_FORM when opening form2 (the one without the menu)
3. Do not use DO_REPLACE when calling the first form, but just use CALL_FORM(’form1′)

All of these workaround might change your application works, so please check which one suits your needs.

For anyone who wants to take this up with Oracle support. The original TAR is logged under number 4648156.992 and the support engineer promised to file a bug. As soon as I have the bug number, I’ll update this entry.

Update 10-oct-2005: Oracle found out what the issue was. It’s having a window with the name ROOT_WINDOW in one of your Forms. The Forms that started out in SQL*Forms 3.0 all have windows with this name. Windows named ROOT_WINDOW get a special treatment in Oracle Forms and that’s what’s causing the problem. Oracle published note 334210.1 on Metalink about the issue. The easy fix is to rename the windows to a different name.

Data on Windows clipboard slows down Forms

Ever since the migration from client/server forms 6i to web-deployed Forms 10g, some users have been complaining about performance. As it turns out this is related to having (large amounts of) data on the Windows clipboard.

Some users were complaining about bad performance when navigating to other fields. For testing I opened up a form with quite a number of fields and started navigating through them using the TAB key. I noticed that the delay was only on navigating to text items. Navigating to pop lists, checkboxes, radio buttons, push buttons and other GUI elements was not an issue.

Some users tend to suffer from this performance decrease more than others. Also, the problem magically disappeared sometimes. Further testing revealed it is due to having large amounts of data on your Windows clipboard. Some of our users are heavily involved in testing and writing manuals. They tend to make quite a number of screenshots using Print Screen. This places a screenshot on the Windows clipboard which can be quite a number of megabytes depending on resolution and color depth.

You can easily reproduce the issue yourself. Open your Windows task manager. On the processes tab, add the PF Delta column (View > Select Columns) to show the number of page faults (swapping). Now open your form and tab through the fields. In the task manager watch your IExplore.exe (or other browser) process. Notice the low number of page faults on navigating. In my case, this was typically below 10.

Now open a text editor and create a large amount (2-4 megabytes) of text. Select all text and copy it to the clipboard. Go back to your form and tab through the fields. When using 2 megabytes of text the PF Delta shot up to 2000 in my case. The larger the amount of data, the higher the paging/swapping will be. 4Mb of data raised the PF Delta to 4000. 8Mb of data raises the PF Delta to 6000. Also notice the high increase in Memory Usage for your process. After this last test it increased to a whopping 140Megabytes! Testing with 16Mb of data raises the PF Delta to 35000. Copying a small amount of data to the clipboard does remove the noticeable delay and reduces the PF Delta to normal figures. Unfortunately the memory usage of your webforms session does remain extremely high (more then 100Megabytes).

There is a limit to all this. Eventually, JInitiator might raise an OutOfMemory error which you can see in the JInitiator Console. When testing with Sun JPI your entire session would eventually crash with a FRM-92050.

Searching through MetaLink made me found bug 3130709. This describes exactly my problem, but is supposed to be solved in Forms 6.0.8.25 in February 2004. I opened an iTAR and that resulted in the same Oracle bug number and a link to Java bug 4790833. That Sun bug was fixed in Java 1.5, but my problem still reproduces in Java v1.5.0_03. If you read the bug text at Sun, you find a reference to bug 4287795 which goes on about having to get the entire content of the clipboard to just know if something is on the clipboard at all.

Today, the support engineer told me that the Forms applet is checking the status of the clipboard to determine whether the Paste entry in the Edit menu should be enabled or disabled. Apparently it is a Sun Java bug that the content of the clipboard is retrieved when just asking for the status of the clipboard. However, Oracle should have worked around that with the fix for bug 3130709. He found a reference to a parameter disableValidateClipboard in the bug description. He suggested adding a parameter disableValidateClipboard=true to my formsweb.cfg but that did not help. Oracle will investigate internally what happened to the bug(fix) and whether it dropped out of 9i/10g. I’ll update this blog entry with any news.

PS. If you found this blog entry because your having performance issues with webforms, you might also want to have a look at the DNS issues we had.

Update 26-jul-2005: Oracle just acknowledged that the fix of Forms 6.0.8.25 somehow was not included in the 10g version. We’ve asked for a one-off which means that development will try to provide a fix for this issue separately for the 10g version. Oracle promised to keep me informed. If they do, I will again update this blog entry.

Update 5-aug-2005: Oracle has openend a new bug 4520121 against Forms 9.0.4.1.0 to also include the fix in Forms 10g. The bug can be monitored through MetaLink. It appears that the final fix has slipped to Forms v11.x, but Oracle will release one off patches for 9.0.4.2.0 and 10.1.2.0.2 per our request.

Update 10-oct-2005: Oracle just released a one-off patch 4599915 for Forms 9.0.4.2.0 that fixes the issue.

Update 29-nov-2005: Oracle just released a one-off patch 4520121 for Forms 10.1.2.0.2 that fixes the issue.

Background of textitems black or pink

We’ve been strugling with a Forms issue for quite a while now. We’ve converted about 300 forms from version 6i to 10g and found out that some of them use a very dark gray (almost black) or pink color as background for some textitems. It occurs for textitems that have no background color set. This shows as “Unspecified” in Forms Builder. When you initially start the Form all looks ok, but after opening and closing some other forms in the application the items suddenly change to this darkgray or pink color.

Quite a number of our Forms originated in SQL*Forms 3.0 and were converted to 4.5, 6i and now to 10g. As it turns out these have a different color pallete than the ones that were created in 6i straight away. It looks like Oracle changed their default color palette along the way. The old forms have exactly the same palette as is now available in the CDE1.PAL file still in the Developer Suite distribution.

I filed a TAR at MetaLink and that resulted in a new bug (4383847). As it turns out, Forms deployed over the web cannot cope with different color palettes in the same session. Oracle says it has something to do with Java restriction. The answer is that you should not use custom palettes. I tried to argue that my palettes aren’t custom but just the default ones of 10 years ago, but that didn’t help.

The steps to reproduce the issue are:

• Start Forms Builder v9.0.4.1.0
• Edit > Preferences
• Set Color Mode to Editable
• Restart Forms Builder
• Save the new form (module1) under the name DEFAULT.FMB
• In the Forms Object Navigator create a new Canvas
• Also from the navigator, create a new Block (do not use wizard)
• Set block property “Database Data Block” to No
• Navigate to the Items node and press the green plus three times
• Select all three new items and set the Canvas property to the created canvas
• Set the Item Type of the second item to “Display Item”
• Set the Item Type of the third item to “Push button”
• Add a When-Button-Pressed trigger to the Push Button with the code call_form('cde1');
• Open the layout editor
• Shift the three items around to fix the overlap
• Save the Form
• Copy the DEFAULT.FMB to CDE1.FMB
• Open the CDE1.FMB in Forms Builder
• Change the Name property of the form to CDE1
• Open the layout editor for this Form
• Select Edit > Import > Color Palette
• Browse to ORACLE_HOME\tools\common90 and import CDE1.PAL
• Change the When-Button-Pressed trigger to call_form('default');
• Save the Form
• Compile both forms using Forms Builder
• Start DEFAULT from Forms Builder
• Notice the correct colors for the text item and display item
• Press the button to start the CDE1 form
• Notice the CDE1 form to open with pink items
• Press the button to start the DEFAULT form
• Notice the DEFAULT form now also opens with the pink items
• Close your webbrowser and go back to Forms Builder
• Now start the CDE1 form first
• Again notice that the first form shows with the correct colors
• Press the button to start the DEFAULT form
• Notice the DEFAULT form opens with very dark gray (almost black) items
• Press the button to start the CDE1 form
• Notice the CDE1 form now also opens with the dark gray items

I already mentioned this problem in a previous blog posting about our Forms conversion. That raised some questions with people, so I decided to document it in this separate blog entry. Also in my quest for a solution to this problem I found several forum questions with the same behaviour of items changing to dark gray, black or pink. Hopefully this blog entry can be helpful to these people.

The only solution provided by Oracle was to manually import the default palette into each and every form. To us, that was not a workable solution. We’re planning on running Forms 6i client/server and WebForms 10g in parallel for a couple of months. Developers continue to work in 6i only and the Forms are converted to 10g when deploying. This means our conversion process has to be repeatable and manually importing a palette is not an option. Unfortunately the JDAPI does not support changing the palette, so we couldn’t automate it that way. The solution we choose was to use the JDAPI to set the background color property of all items where it is not set. I’m setting the background color to “r100g100b100″. In effect, this is just white but with a different name. This makes it possible to recognize the items that were changed by the JDAPI. Whenever Oracle fixes the palette-bug, I might decide to change these items back to Unspecified to get really transparent items. Now I’m having white items on a non-white canvas, which isn’t what the users ask for.

If you’re struggling with white background of readonly items (both textitems without modification rights and display items), please read note 305563.1 on MetaLink. It describes a applet parameter readOnlyBackground. You can set this to TRUE to force all read-only items to use the canvas color as background. A more usefull setting is AUTO which uses the canvas color as background for readonly items that do not have a specific background set. If a readonly item does have a background color set (e.g. a current record visual attribute) then that color is used when using AUTO.

Update 29-jul-2005: Also see this blog entry of Duncan Mills which describes changing the color palette of Forms and in his introduction mentions the issue with black-on-black items.

Faulty DNS causes lousy performance WebForms

We’re currently migrating from Forms 6i client/server (deployed via Citrix) to Forms 10g via the Application Server. Since the beginning we had issues with very slow performance. Initial startup of the application could take as long as 2-3 minutes. But even when you’re logged in, performance was far from optimal. The first time you opened a menu it took 10-20 seconds.

After days of puzzling we finally found it. A network trace of the workstations showed they were performing numerous DNS lookups. We connect to the application server using HTTPS via a normal web-proxy. So, the URL is something like https://appserver.example.com/forms90/f90servlet. A network trace showed that the workstation is doing a DNS query for “appserver.example.com”. This is not necessary, because contacting the application server is left to the proxy-server and the workstation never needs to make direct contact with the application server.

In our case, the DNS lookup caused huge delays. It turned out this was caused by a faulty DNS. Our internal network doesn’t heavily depend on DNS and so we never ran into this issue before. The DNS server is a standard out-of-the-box installation. This means it has the 13 default root servers configured. This is fine for an internet-connected DNS server, but does not work for internal DNS servers. Our internal-only DNS server cannot contact the 13 root servers.

Back to the workstation. It is sending a DNS request for “appserver.example.com” to our internal DNS server. That server cannot handle the request and will start asking the root-servers. Since it cannot contact these servers this will lead to considerable delays.

It could very well be caused by a bug in Sun’s Java or be related to a bug. If you search for “DNS lookup proxy” in Sun’s bug database you find a number of hits and not all of them have been solved.

The easiest way to correct the problem is making your DNS server respond to the query instantly. You can either do this by letting the DNS server contact the root-servers on the internet, or just add a record for your application server to the DNS server, or remove all root-servers from your DNS server. The last solution will make your DNS server respond to the request instantly. It will just respond with a “Non-existent domain” response, just as it would when you ask any DNS server for “example.invalid.domain”.

With all testing we also found another potential DNS issue. In our browser the proxy server is configured by IP address and not by hostname. For some reason, Sun’s JPI then performs a reverse DNS lookup for this IP address to get the hostname. It then performs another DNS lookup to get the IP address for this hostname. From information in Sun’s bug database I understand this is for extra security and to prevent DNS spoofing attacks. Just keep this in mind, and make sure the IP address does resolve to a hostname and the hostname resolves to an IP address again. It did in our case, so we didn’t have a real issue here.

When trying to resolve the issue, we’ve spend quite some time browsing OTN forums and MetaLink. I couldn’t really find an answer there, but did find numerous other Oracle users with similar problems. I hope this blog entry can help them.

Update: As stated before, if you specified your proxy server as an IP address (and not as hostname), Sun’s JPI performs a reverse lookup of this IP address to hostname and then resolves this hostname back to IP address. We just found out that when your DNS server responds to the initial reverse DNS lookup with a response like ‘Non-existent domain’, your Windows client will then revert to a Netbios lookup of the IP address which takes even more time. The lesson learned is that when you want to use Sun’s JPI and you specify the proxy server as an IP address, you have to make sure there is a proper PTR record setup in your DNS server and that the name resulting from the PTR record resolves back to a valid A record.