ESCAPE VELOCITY

1 - 1. No, not regularly, but have done it in the past. I most often do this after the initial save and use subforms/child forms for changing what is shown to users during different steps in the workflow. I do use Form Fomulas to display documents using different forms, but usually it's for read only purposes.

2. If they know what the issue is and are simply choosing not to fix it, that is totally not cool. It shows a real lack of concern for the development community, yet again.

3. No, but I will keep an eye out.

BTW, please fix the CSS for the comments form in IE7 before it drives me insane!

2 - Its not something I normally do. I see it as a hack where a subform should have been used. That said, it is a perfectly legitimate action to take as a developer and is a fully supported operation. The bug should be fixed.

3 - 1. It's been years. I vaguely remember having issues with it, so I went down another path, subforms, and haven't looked back. That's one of the great things about Notes dev, often there's more than one way to solve a design issue.

Any knowledge as to how big a performance drop-off is without this option checked?

2. Yes, but only because they don't appear to have asked anyone but their internal Notes dev's. As Mary Beth has shown, those days should be behind us.

3. I don't know.

4 - 1 - No. Well, I try not to anymore since I had my own "A-HA!" moment. See, changing the "form" NotesItem just isn't a requirement if you architect your applications to have context sensitive UIs. Display a UI based on client entry point, functional entry point, etc. but not based on the actual value in this NotesItem. Stick with "document", "response", etc. and let QueryOpen-maintained logic display a SaveOptions=0 UI Form that writes data changes back to a Notes Logic Broker which updates the backend document(s). I could be wrong though... it's been known to happen.

2 - Total oversight... not done through any malicious intent but "an oversight" by definition. Mind you, unless this breaks XPages, I doubt you'll see a fix high on anyones priority list anytime soon.

3 - No specific names come to mind, but I know that I *have* seen it "in the field". Each time I do, I shudder... I'll keep a close eye out and hit ya up with product names as I see them.

5 - 1. Yes, I have worked on an number of applications that involved changing a form name for one reason or another.

2. Yes, I think it's definitely something that should be fixed by IBM.

3. None that I know of.

6 - Well, it looks like the preliminary responses go against my expectations, so perhaps I'll need to publish a mea culpa.

Still, I'm surprised. "Proposal" to "Project"; "Prospect" to "Customer"; "Candidate" to "Employee"; "Trouble Ticket" to "Bug Report" -- this isn't embedded in your own applications? I've been doing this for years, and never for a nano-second would have considered it a hack. Just smart programming.

And Toohey, you don't write everything to a "logic broker." That's a recent idea of yours, and the fact that it's a good one doesn't change the hundreds of forms you've already put into production. Puh-lease.

7 - 1. If I have, I can't remember it right now, but it doesn't seem that left field to me so I might have, I wouldn't be embarrassed about it anyway.

2. Grievous might be a little strong, but plain oversight is a little light. A bug is a bug, introducing one is not OK just because you personally aren't aware of any situations where it will *definitely* cause problems.

3. Nope.

"Proposal" to "Project" - Status field of the Task form?
"Prospect" to "Customer" - Status field of the Contact form?
"Candidate" to "Employee" - Status field of the Employee form?

I try to map my documents to real world entities as much as possible, the basicer the betterer, a shape shifter usually means look at it from higher up. Not that I would call it a hack by any means to change the Form field value of a document, heck, a document doesn't even strictly need a Form field! It's just a field like any other isn't it? I've never seen anything that says "if you set the Form field, *DO* *NOT* change it afterwards", and I would give my head a shake if I did.

I'm with you man. Sure there are many ways to get something done in Notes, but I wouldn't want to be the one rewriting applications to use the way that IBM didn't cripple

8 - @7 - Some great examples...

You say "Proposal" to "Project" is a change of status. I can't disagree. I would change the status. I might also change the resulting form. Why the form? To get a different set of fields.

Why would I not just present the fields wit elaborate hide-whens? Because they're a maintenance nightmare.

Why not just use computed subforms? Ummmmm... because I like to write and reuse Lotusscript classes? When combined with subforms, Global object instantiations are one of the quickest ways to make an apparently smart app incredibly slow.

The law of leaky abstractions, indeed.

And one important remedy to leaky abstractions is that the platform provide a work around. With this issue, there really isn't one.

9 - A few points of clarification...

1. This issue is documented here: { Link }

2. The workflow described may not be rare, but known occurrences of this SPR are. I would encourage anyone who is seeing this issue to log a PMR with Support or email me with the details.

3. There is not currently a fix for the issue, which is why it is not as simple as bundling this in with other things to test. We do view this as a serious problem and continue to evaluate ways to fix it.

10 - 1. Sometimes though I tend to shy away from it.
2. This has to be fixed ASAP, because it has a high potential to suprise developers. Imagine a project manager who has to put an application into production because of this bug. The time to change the application-code to one without changing the form field is a) probably quite long and b) difficult to predict. These are the 2 things project managers definitevly hate for good reason.
3. no.

11 - @9 - Chad, thanks for the clarification. According to the documentation you linked to the problem has been fixed for version 8, right?

12 - 1. Our main apps use this frequently. In different databases. This was never a problem before 6.

2. Fix ASAP. A bug in the view selection code? Unfixed for years? Seriously? I am lost for words. Especially since according to the first forum link in the amin article this bug did not exist in R5. This smells of the kind of 'fix' that was done to the web server scheduling algorithm in 6 (see <a http="{ Link } look for HttpQueueMethod=2 in the comments)

3. Not that I know of.

So: I really would like this to be fixed. If at all possible, also in the older streams, without the ODS structure change. Please do not make it a NotesIni parameter. This bug is defenitely not a feature.

As for the request for PMRs: I understand they would be nice. I will if I see it happening. Is it a possibility to compare R5 and R6 code streams?

13 - Ummm,
I hate to be a nitpicker, because your post is spot-on.

The option is "Optimize document table map", NOT "Optimize document table bitmap".

-Devin.

14 - Gah. Forgot to answer your questions.

1) No, not personally. However I've seen it done in many, many applications and have never considered it to be a bad thing. In some instances, it is the perfect solution to a rather thorny problem.

2) YES!!! The view is the basis for all nsf databases. You cannot have an nsf without one. It is the number one most important "thing" there is. A bug that causes some documents to appear in a view when they are excluded by the view selection formula, no matter how rare, is a FLIPPEN HUGE BAD THING! Failure to fix this goes way beyond oversight. This needs to be fixed right now.

3) I can't remember right now, but for some reason the old Redding Bicycle Co. application (from AD2 course ware) comes to mind. I don't have a copy at hand to check right now. Although I guess that is a red herring, because this was pre R6.

-Devin.

15 - @13 - I was writing the correction when you replied!

@All - Read the update to find an out-of-the-box template that uses WebQuerySave events to reset a Form value. It's a famous one. And one that famously has problems!

16 - I'm the one that has spent the past few years helping IBM track this one down, and I'm the one posted about it on the forums.

It sounds like a couple of people didn't see the link in the article to the forum post so here it is:

{ Link }

The performance hit is potentially huge. Read the bottom of the forum post for an example.

If you're interested in this getting fixed please open a PMR and request to have SPR# DCOE6N82TP fixed. And please do it soon, as an ODS change is required to fix the problem and now is a perfect time to address it since the ODS for 8.5 is switching anyway. If IBM asks you for a reproducible example and you don't have one (which I'm guessing you don't), simply say that you want to be able to use the performance-boosting "Optimize Document Table Map" switch but can't due to this known, documented problem.

To be fair, IBM didn't fully understand the problem until a couple of weeks ago (it has taken years to fully reproduce). And they agree that it's nasty. It's a pretty high priority internally, but is currently "no plans to fix" without more weight.

Also, the technote referenced by Chad is not completely accurate. Specifically:

- The problem is reproducible in 8.0, 8.0.1, and 8.5 beta 1.

- Chad is correct, *known* occurrences of this SPR are ultra-rare. Once the problem occurs, it simply looks like a corrupt view index. One you can fix with Shift-F9, Updall, setting the view index to discard, or any other number of "blow it away and rebuild" techniques.

- The technote states that the problem "does not occur if the document is being modified by the Notes client." I would place money on the fact that IBM doesn't know that definitively, as this thing has been an absolute bear to reproduce and is very rare and intermittent. We believe it *has* happened with code run in the the Notes client, we just haven't been able to get a DB in a pre-broken state to IBM for reproduction. If IBM does know definitely that the problem *cannot* happen with the Notes client, they should doc that.

17 - @16 - The SPR is dated 2006-07-28. That's just shy of two years.

I'm honestly stunned that this isn't on the list of "NSF reliability" issues that the Domino server team has as a main priority for 8.5.

The fact that IBM claims it only happens via web updates against the Form field is precisely what leads me to believe that the issue is one of testing rather than coding. If you know that much detail about where the problem happens, tracing the code and coming up with a solution is usually not too difficult.

18 - @17 - I've seen the problem since early 2004 and wasn't able to submit a pre-broken-state immediately-reproducable DB until Feb 2006.

I'm shocked this isn't on the 8.5.x ticket, too. I've found numerous things over the years that are "just plain broken", and most of them can be coded around or have been fixed promptly after I've opened the PMR. In particular, the Web Server team rocks hardcore at squishing bugs. But this one's a doozy. A database with potentially flawed indexes simply isn't much of a database.

19 - By the way, the front page of this blog shows 16 comments for this entry, when there's 18 as I type this.

Hopefully it's not some kind of bug with Form fields and view indices!

20 - @19 - Nope. It's a flaw with the way Blogsphere tracks comment counts. When I did the update, it generated a repl conflict. The one with the proper comment count lost.

I'm just too lazy to code up a fix. (Each additional comment should cause a recount of ALL comments, not just increment by one.)

21 - I've had senior technical people at IBM tell me that Optimize document table map doesn't actually do anything to improve performance and in some cases may make an application slower. So even though it is not uncommon for my applications to support a change in Form names it is not a problem. Please let me know if your experience shows that this option does actually make a difference to performance!

22 - 1) Yes, but I quit doing it when I started having problems. I remember I spent a few weeks going back and forth with Lotus Support before we tried creating multiple documents instead of changing the Form field. The problem went away and I just chalked it up as yet another Notes quirk. I have never done web development so this was on the Notes client.

2) Yeah, it's grievous. To me this idea of "yeah, we know it's broken and we even know how to fix it, but since not enough people have complained we're going to ignore it" borders on unethical. If you know of a problem and you know how to fix it, DO IT.

23 - @6:
All I got from that was "Toohey, that's a good idea there!"
But I hear ya - sure there are tons of apps that I have in production today that use this approach - the "graduation" of a document through a workflow via different "form" NotesItem contents.
But doesn't this flaw more define the need to change our everyday development practices instead of relying on the vendor to fix the issue? Of course, there's the small "I upgrade, thus breaking 500 production apps" sticking point there... but maybe they're trying to help out the "professional services" business partner channels with contracts for application re-writes and bug-fixes through re-development.
Of course, if a company is going to have to redevelop their applications due to such an issue... maybe they'll look to different vendor solutions to get the job done where they perceive said vendor addressing their architecture bugs...
Um... so yeah, I'd think that this really needs to get onto someone's task list ASAP to address while we application developers also work towards deprecating this approach (and substitute a "good" one ).

24 - @21 - Those senior technical people are wrong. Try it yourself:

1. Make a new DB with 100,000 docs. Set 500 to form "A". Set the remaining 99,500 to form "B".
2. Build a view with "SELECT Form = A". Note how long it takes to build the index when you first open it.
3. Delete the view.
4. Toggle "Optimize Document Table Map".
5. Compact the DB using a copy-style compact so the setting takes effect.
6. Repeat step #2.

The view will build MUCH faster (over 100x) with the setting on than with it off. With the setting on, Domino can quickly disregard all of the "B" documents. With if off, it's combing through all 99,500 of them just to decide that they shouldn't be in the view.

This affects view updates and NotesDatabase.Search() calls. The more docs you have, and the smaller set you actually work with, the more apparent the boost becomes.

It's interesting to note that this sort of on-disk indexing is similar to the indexing available by the world's fastest relational databases. It would be incredible if Notes could support this indexing on any field(s) we developers defined, though that would understandably be very difficult to develop as a feature to a schemaless database.

I'm guessing that awhile back somebody at then-Iris said "Well, *everything* has a form field. And it's always text. And 99% of all views look at that. I guess we could add some optimization code..." and voila - this feature was born.

25 - @23 - The problem here is that the feature is broken. You don't need to recode your apps -- you can simply turn off the feature and *poof*, problem's gone.

If you recode around the problem you are simply forgoing the performance boost in the first place, which is the same as effectively turning off the feature. The indexer will happily "crack open" each and every document against each and every view update needed, "Sure shootin'". That's how R4 did it.

If you want to see another example of the performance boost repeat my steps above, but run an agent that sets a field on the 99,500 form "B" documents. Then open form "A"s view. The view will open lightning quick because it knows that the 99,500 docs you updated don't belong there in the first place.

This is exactly the kind of bug that would get crushed ASAP in the open source world.

26 - @11 - No. The issue is not fixed in R8. I have updated the Technote accordingly.

27 - Hi Nathan:

re: In other words, "this bug doesn't affect our templates, so it's not a priority."

That is not the mentality of anyone here who is near this problem. You do a disservice to some really good people to project motives like that.

We know the problem and a) its rarity, b) lack of customer weight (aka squeaky wheel syndrome) and c) an available workaround (i.e. turn it off) are the reasons it hasn't been addressed. Believe it or not, there are issues that our customers find more important. Also, believe it or not, this is not THAT deep of a problem (read "we know how to fix it"). It will cost a few CPU cycles during update, that's all.

You can't change or really contest a) (argue if you like) and you're saying c) doesn't help you (quite believable, the form bitmap index is a huge win for form-based selection), so change b) and it will get attention. Open a PMR and yell till it gets addressed. I think you know that's how that works.

Once it's on our radar screen, we'll nail this, that's a promise.

John Curtis
IBM Domino Database Tech Lead

28 - John, thanks for the reply. But honestly, I don't understand it at all. You've basically said I'm doing a disservice to good people, because I've said that they won't solve a problem they know how to solve simply because there's not enough noise or internal pain about said problem, and then you go on to say that 1) you do indeed know how to fix it; and 2) you need more PMRs filed in order to fix it.

quod erat demonstrandum!

I don't mean anything personal by this. I know many fine people working on the Notes/Domino product line. It's rare that I interact with anyone on that team that I don't find a treasure on a personal basis. It's the IBM process that's the problem, not the individuals. I only even mention Chad's name because he was the poor messenger here. But he's delivering the institutional message.

I'll obviously do whatever I can to get PMRs submitted, but honestly, do you really think it's fair to put the onus on your customers to go request a fix for a bug you already know about? Particularly one that will be written off by almost every single Lotus professional as "oh, the index is corrupted" and never followed up?

I understand you guys are in a process bind, but this customer weighting process doesn't seem particularly agile to me. It's frustrating and depersonalizing from a customer standpoint. And that has nothing to do with all the really good people working on the product, everyone of whom has my respect.

The purpose of this post is to get it on your radar. As is the purpose of delving into your own templates, and digging up a place where you yourselves are reassigning the Form value, on what is probably the most infamously unreliable template in all the Yellowverse. How many PMRs do you guys have on Resource Reservations that are related to "index corruptions" or "disappearing reservations?"

29 - Well, this issue got me looking into an application recently put into production. Sure enough, it looks like it's impacted by this exact bug, and it does NOT involve the HTTP process. The functionality uses a scheduled agent that imports records from an Oracle database. In order to cut down on the time necessary to keep the connection open to Oracle the records are saved with a temporary form name, then a follow-up process gets those documents from a view that uses a "Form =" selection and sets the permanent form name and does a compute with form. Inspecting that view I found it contained documents that had indeed been processed, had their form name changed and should no longer be appearing in the view - but they are! Doing a Shift-F9 corrected the problem (for the moment). Now I need to re-consider the approach on imported those records.

30 - @27 - Thanks for reading, John, and thanks for the attention given to this particular SPR as well. That goes to everyone involved, here in the community and at IBM.

Regarding: "...a) its rarity, b) lack of customer weight... c) an available workaround..."

What Nathan and I are arguing is that (b) is likely much higher than reported. And because of (a), I argue that most people being bitten by it simply don't know that they are, or else (b) *would* be higher. Based off of Nathan's research, potentially any PMR/SPR involving one of these...

- "corrupt view"
- "performing a compact fixed the problem"
- "busytime strangeness"
- "Shift-F9"
- "Ctrl-Shift-F9"

...with any of these...

- "NAB"
- "custom app"
- "Resource Reservation"

...against any version of Domino running ODS 43 or higher could have been affected by this bug in the first place. That's ND 6.0 and up (but I'm sure you know that ) which means nearly 6 years of this bug causing problems.

Those facts *should* add more weight to this SPR, unless the above symptoms occur less frequently than we all believe.

Hopefully everybody reading this and saying "YES!" is opening tickets to get this addressed. Because it involves an ODS change I don't see it getting resolved for a long time unless it makes it into 8.5.0.

31 - Hi,

I don't mean anything personal by this either, so I guess we agree

1. People don't often change form fields on Domino documents with the option on. If they did (ok, sure, if *we* did), I think we all know this would not still be an issue.
2. The process is what the process is. With this many customers, we need some mechanism to prioritize things and it's not likely to be a browse of the blogosphere.

So, sure, lobby on.

-John Curtis

32 - @31 - John, just to be clear: it's not "if IBM did." You do. In the Resource Reservations template. There are several kinds of forms that are switched to "reservations" by WebQuerySave agents.

Also, you do it on the NAB in the Connections form. It's just that the one view that looks at the Form value in the Connections documents is inclusive of all form types for connection docs, so the view would never appear corrupted because the selection set would include the document in either condition.

But I find myself wondering what Russ Holden might say to the idea that the "optimize document table map" flag, which could dramatically speed up every directory and every mail database on a deployed on a Domino server, isn't actually being used by IBM? I know Russ is an absolute fanatic about wringing performance out of the server, and the truth is, when I look at the combination of this flag, and the number of base templates that *don't* use Form= in the selection criteria, it seems like there's a whole lot of performance left on the table.

I'm not saying that to invoke him on the matter or anything. It just seems like such an easy win for performance tuning. Faster indexing on directories and faster indexing on calendars seems like it'd be something that IBM would really want to pursue. (And yes, I realize that those kinds of changes aren't likely to happen on the directory template in the 8.5.0 timeframe.)

Heck, the bug itself wouldn't even affect you substantially. Chad does have it right that NAB forms *almost never* change. It's not like there's a lot of server connection documents out there that are anything but "Local Area Network" anyway.

33 - I used to design apps that changed the Form field all the time. In R3, before we had all the goodness of things like subforms, hide-whens, and LotusScript, changing the Form field was simply the only way to write even a moderately complex application. Those were also the days of servers with 500 MB of storage, total (yes, that's an "M", not a "G", and really 500 MB was what you had for an R3 server if you were lucky!), so creating additional docs as workflow state changed was just not something you wanted to do.

I don't know when I stopped. It had nothing to do with this bug, of course, and I'm sure it wasn't right after R4 shipped. I'm sure it wasn't until I had servers with bigger disks, not to mention bigger practical and coded limits on database size, before I decided that creating new documents was ok -- and had the added benefit of preserving previous docs as a record of the workflow. It took some time for it to sink in that there were better ways, but eventually my coding idioms changed. I can't remember the last time I designed an app that changed the Form field.

That said, the bug is significant and I think it should be fixed... but I live in an environment similar to IBM's, so I have to give their position a great deal of respect. Where I work, engineers also don't necessarily get to fix every bug they know how to fix, or want to fix, whenever they feel like it. There is a cost, and often a lot of risk, inherent in every fix, and that can't be ignored. An assessment of probability that a bug will affect customers, and of the impact on customer if it does, has to be made. An analysis of risk of side-effects, and of delays to release schedules is also done. If a bug is over a certain threshold of probability and impact, and if no workaround exists, and if the risk is acceptable, then the bug gets fixed... period. Otherwise, it may not be fixed until customer pressure forces the issue. This seems self-defeating. It seems like it is designed to maximize the frustration of customers and engineers. It's perverse that in the name of protecting product quality, some known and understood quality problems are deliberately not fixed! Unfortunately, I'm not aware of a better system, so I understand what IBM is saying.

Nevertheless, I disagree with the call on this particular bug. In my job, when I assess the impact of a bug, for the top level of impact I am concerned with whether the bug causes the system to crash, hang, or irretrievably lose data; and at the next level I am concerned with (amongst other things) whether a bug causes a feature to be completely unusable. Bugs at these two levels are considered must-be-fixed. IBM should be concerned with all of these same impacts, but due to the nature of Notes and Domino as an application platform they should also be concerned with the impact of a bug on applications. Namely, IBM should be concerned with whether the bug is likely to cause applications to fail completely or lose data for no apparent reason... and IMHO this bug is a prime candidate for causing exactly that. And since the only workaround effectively declares that a product feature is unusable, the workaround does not drop the impact of the problem below the second level of impact, and therefore it should be a must-be-fixed bug unless the cost and risk of the fix is absolutely unacceptable. I know I'm being presumptuous in saying this without knowing the real nitty-gritty details of the bug, but this doesn't have the feel of a high cost/high risk fix, so I don't think it calls for a pile-up of customer PMRs to get it fixed.

34 - @33 :
Quotes : "In my job, when I assess the impact of a bug, for the top level of impact I am concerned with whether the bug causes the system to crash, hang, or irretrievably lose data; and at the next level I am concerned with (amongst other things) whether a bug causes a feature to be completely unusable."

The higher level is ( for me ) : security. After this point OK : crash, data loss, ...

The bug mentionned in this blog entry, has not an high impact for me. It has maybe for Domino administrator when they search to optimize application speed, but it's not a "flaw in security", "a bug that crash, hang a server", ... It "only corrupt" views... and view corruption, we know how to solve this kind of problem....

It's not a bug that "corrupt form design" for example ... that's a lot more frustrating if you have passed your day to develop and have only a daily backup....

That's my personal opinion.

35 - It "only corrupt" views... and view corruption, we know how to solve this kind of problem....

But do you know how to identify it?

This wouldn't be a big deal if it were easy to tell that the view was showing inaccurate information. The rebuild is fairly easy to perform. The problem is, how do you figure out that the right information isn't in the view in the first place?

36 - >> But do you know how to identify it?
>> This wouldn't be a big deal if it were easy to tell that the view was showing inaccurate information.

Ok fine, I see now what you mean by that. We see it with the complaints of the customers. ( that's what you wanted to hear ).

If it's a view that is used by scheduled Agents, they will be some problems to detect it. You are right.

In that case, it's a degrading service, ok it's not so big than a security flaw or a server hang/crash, but it's kinda a big deal.