Database Forum / Oracle / Oracle Server / December 2003
Deadly sins againts database performance/scalability
|
|
Thread rating:  |
Dusan Bolek - 25 Nov 2003 16:03 GMT Hello,
I'm just creating some document how developers should/must not behave when dealing with Oracle database. I written down some basic rules and came with idea what's the most stupid thing that you can do to your database. I have two on mind, first one is not using BIND variables and the second one is using COMMIT in a loop statement. However, there will be more of these. So I have question to you here. What's the most stupid, but in real world foundable, mistakes in development applications using Oracle datatabase?
-- with regards
Dusan Bolek
Norman Dunbar - 25 Nov 2003 16:35 GMT > Hello, <SNIP>
> So I have question to you here. What's the most stupid, but in real world > foundable, mistakes in development applications using Oracle datatabase? 1. Not taking backups :o) 2. Adding indexes all over the place for performance reasons. 3. No (proper) change control procedures. 4. Exporting from the database and defaulting COMPRESS to Y, even worse, exporting with COMPRESS=Y explicitly. 5. Allowing developers to create obbjects in a tablespace with a different extent size to the tablespace defaults. 6. Having numerous indexes (on the same table) which could be replaced by one or two carefully thought out indexes. (you know, indexes on A and B and A,B and C and A,B,C and B,C ....) 7. Putting the ref int checks in the application instead of the database. 8. Ditto, but keeping business rules in the app instead of the db. 9. Using standalone procs and functions instead of packages. 10. When actually writing packages, not separating the PACKAGE specs from the PACKAGE BODY so every time you change the code in a body, you re-create the spec as well and invalidate everything that depends upon it. 11. Using a table to hold a sequence number is a bad one. You know, read the number and lock it, increment it and write it back - then commit at some point in the far future. Even committing ASAP is a major problem as soon as you get more than 5 or so users all doing the same thing. 12. Even worse, having a single table for all the sequence numbers in the system. 13. Creating 'temporary' tables to hold intermediate results MS SQL Server style. 14. Converting triggers from another db system into Oracle and discovering that the other system allows you to update rows in the table that has triggered the trigger leading to mutating table errors on Oracle (which IMHO does it correctly !) 15. Not enough SQL reuse.
I'm sure that there are more :o)
Cheers, Norm.
 Signature Delete the obvious bit from my email address to reply by email.
Norman Dunbar - 26 Nov 2003 08:51 GMT <BIG SNIP>
Sorry folks, looks like my formatting went the same way as Australias chance of winning the RWC - it all went wrong at the end !
Cheers, Norm.
 Signature Delete the obvious bit from my email address to reply by email.
Niall Litchfield - 25 Nov 2003 16:49 GMT > Hello, > [quoted text clipped - 8 lines] > world foundable, mistakes in development applications using Oracle > datatabase? No primary key - rare No foreign keys - common No bind variables Create intermediate tables as often as possible and then drop them when done with. Use PL/SQL instead of SQL No Bind variable. Use Java SP instead of PL/SQL Use EJB instead of the Database. Use Net instead of the Database No bind variables . Avoid array fetch at all costs. Parse repeatedly. Test on your laptop with a maximum of 5 rows per table. Don't test at all, that is what pilots are for. nO binD variableS. Return all of the table to your client and filter on the client to avoid loading the server. Never document anything - noone will read it. Never have a database model - that is just for classes. no bind variables
:(
 Signature Niall Litchfield Oracle DBA Audit Commission UK
Daniel Morgan - 25 Nov 2003 20:44 GMT >>Hello, >> [quoted text clipped - 32 lines] > > :( You forgot No Bind Variables. ;-)
 Signature Daniel Morgan http://www.outreach.washington.edu/ext/certificates/oad/oad_crs.asp http://www.outreach.washington.edu/ext/certificates/aoa/aoa_crs.asp damorgan@x.washington.edu (replace 'x' with a 'u' to reply)
Jeff - 26 Nov 2003 18:15 GMT What are bind variables?
;-)
>> Hello, >> [quoted text clipped - 32 lines] > >:( Norman Dunbar - 27 Nov 2003 08:52 GMT > What are bind variables? > > ;-) Hi Jeff,
who mentioned bind variables then. I don't think it was Niall.
:o) Cheers, Norm.
 Signature Delete the obvious bit from my email address to reply by email.
Andy Spaven - 27 Nov 2003 15:52 GMT Hi all
I want to laugh about the bind variables posts but I might cry given my current employers attitude - All I can say is thank god for cursor_sharing=force :-))
Andy
> > What are bind variables? > > [quoted text clipped - 7 lines] > Cheers, > Norm. Niall Litchfield - 27 Nov 2003 16:10 GMT Actually I'm beginning to regret that post now.
As Joel rightly says there is a real danger in creating a list like that one. In my case my post might just have suggested that one should always use bind variables (I believe you could read it that way :) ) and this is of course untrue, one should always use them, except where they are inappropriate.
 Signature Niall Litchfield Oracle DBA Audit Commission UK
> > > What are bind variables? [quoted text clipped - 8 lines] > Cheers, > Norm. Daniel Morgan - 27 Nov 2003 18:28 GMT > Actually I'm beginning to regret that post now. > [quoted text clipped - 3 lines] > course untrue, one should always use them, except where they are > inappropriate. And they are in appropriate ... when?
 Signature Daniel Morgan http://www.outreach.washington.edu/ext/certificates/oad/oad_crs.asp http://www.outreach.washington.edu/ext/certificates/aoa/aoa_crs.asp damorgan@x.washington.edu (replace 'x' with a 'u' to reply)
Niall Litchfield - 28 Nov 2003 08:43 GMT Where you have skewed data would be the classic example.
select employee_name,salary from emp where dept='FACTORY FLOOR';
might be expected to run optimally with a FTS.
select employee_name,salary from emp where dept='CEO OFFICE';
probably would best use an index scan. Assuming a reasonable proportion of workers to chief execs :(
if you issue
select employee_name,salary from emp where dept=:b1;
which are you going to want?
 Signature Niall Litchfield Oracle DBA Audit Commission UK
-- Niall Litchfield Oracle DBA Audit Commission UK
> > > Actually I'm beginning to regret that post now. [quoted text clipped - 6 lines] > > And they are in appropriate ... when? Jonathan Lewis - 29 Nov 2003 08:52 GMT Answer in-line
 Signature Regards
Jonathan Lewis http://www.jlcomp.demon.co.uk
The educated person is not the person who can answer the questions, but the person who can question the answers -- T. Schick Jr
One-day tutorials: http://www.jlcomp.demon.co.uk/tutorial.html
Three-day seminar: see http://www.jlcomp.demon.co.uk/seminar.html ____UK___November
The Co-operative Oracle Users' FAQ http://www.jlcomp.demon.co.uk/faq/ind_faq.html
> > Actually I'm beginning to regret that post now. > > [quoted text clipped - 5 lines] > > And they are in appropriate ... when? DSS systems where the queries are few and large, and the optimiser's need for precise inputs is more important than a (relatively) minor increase in CPU and latch costs on parse/optimise.
Ryan Gaffuri - 01 Dec 2003 18:57 GMT > Answer in-line > [quoted text clipped - 13 lines] > is more important than a (relatively) minor increase > in CPU and latch costs on parse/optimise. I also disgree that you should never create intermediate tables and drop them. I think this is taken to an extreme. Now in a high transaction system you should not because it causes latch contention and affects throughput.
However, in a DSS system or for nightly batch processes, Ive found VERY large improvements in performance by doing create table as nologging in paralel. Its much faster than inserting into a global temp table. you dont have to worry about throughput if your looking at 5-10 users on the system max. Now you need to document it well(which most people dont do), so people know what to alter if you need to scale to more users. If I only have a certain window of time to get a series of batch processes done, Ill use temp tables like this. I find they are very useful when working with remote databases. You do a quick create table as to get just the data you need, then do your joins to the local table, so your not pushing data across a DB link for 10 different queries.
However, this will not scale to alot of users. So it needs to be well documented.
Daniel Morgan - 02 Dec 2003 01:26 GMT >>Answer in-line >> [quoted text clipped - 36 lines] > However, this will not scale to alot of users. So it needs to be well > documented. If you ever want to have some fun do the following:
CREATE TABLE test (testcol VARCHAR2(20)) NOLOGGING; ALTER SESSION SET EVENTS '10046 trace name context forever, level 12'; INSERT INTO test VALUES ('ABC'); DROP TABLE test; ALTER SESSION SET sql_trace = FALSE;
Then ...
CREATE GLOBAL TEMPORARY TABLE test (testcol VARCHAR2(20)) ON COMMIT DELETE ROWS;
ALTER SESSION SET EVENTS '10046 trace name context forever, level 12'; INSERT INTO test VALUES ('ABC'); COMMIT; ALTER SESSION SET sql_trace = FALSE;
Run the trace files through TKPROF.
Take a good look at what Oracle does to drop that table. It is not exactly a thing of beauty. Necessary ... veryt expensive.
 Signature Daniel Morgan http://www.outreach.washington.edu/ext/certificates/oad/oad_crs.asp http://www.outreach.washington.edu/ext/certificates/aoa/aoa_crs.asp damorgan@x.washington.edu (replace 'x' with a 'u' to reply)
Ryan Gaffuri - 02 Dec 2003 18:08 GMT > >>Answer in-line > >> [quoted text clipped - 59 lines] > Take a good look at what Oracle does to drop that table. It is not > exactly a thing of beauty. Necessary ... veryt expensive. i know what it does. Ive traced it. Ive also noticed that what it does causes contention for resources and affects throughput. When doing batch processing your often not concerned with throughput. Your concerned with response time. That resource contention is not much of a concern when you do not have to worry about alot of users. when your primary concern is getting the batch done quickly.
Also, alot of people(including us) are moving to shared storage where you run multiple servers on one storage system. In this case physical I/Os become important. Since you may have alot of servers hitting one big pipe. I found that create table as nologging significantly reduces physical i/o over inserting into a global temporary table.
I have found that tuning for throughput and tuning for response time can lead to somewhat different paths in some circumstances.
as I have said before this process will scale up in terms of more data, etc... involved but will not scale for many, many concurrent users. If you need to balance batch processes and concurrency then this would not be appropriate.
Daniel Morgan - 02 Dec 2003 20:26 GMT >>>>Answer in-line >>>> [quoted text clipped - 80 lines] > users. If you need to balance batch processes and concurrency then > this would not be appropriate. And you are, I presume, using the APPEND hint.
 Signature Daniel Morgan http://www.outreach.washington.edu/ext/certificates/oad/oad_crs.asp http://www.outreach.washington.edu/ext/certificates/aoa/aoa_crs.asp damorgan@x.washington.edu (replace 'x' with a 'u' to reply)
Vladimir Andreev - 08 Dec 2003 12:18 GMT > And they are in appropriate ... when? -- When they hide important information from the CBO: * low distinct cardinality columns with uneven distribution of the frequencies. In such cases, you create histograms and, ideally, use literals in predicates. That's somehow addressed by Oracle since 9.0.2 with the bind variable peeking feature, but still... it only has an effect on the hard parse, since binding a different value does not invalidate the parsed cursor (that's the whole point...)
* when using list or range partitioning Using literals allows the CBO to prune partitions at parse time, rather than at execute time
Cheers, Flado
Joe Weinstein - 25 Nov 2003 17:30 GMT > Hello, > I'm just creating some document how developers should/must not behave [quoted text clipped - 7 lines] > world foundable, mistakes in development applications using Oracle > datatabase? The number one performance rule for Oracle or any other RDBMS is "Don't Move the Data". The biggest sin is treating the DBMS as a dumb filing system, and pulling huge wads of raw data out into a client to massage and return row-by-row or digest into a report. DBMSes are designed for efficient set-based processing. Try to extract only the data needed to display. For any data extracted that is non-volatile or under the total control of the one client application, to cache retrieved data in a long-running client (or middle tier) to avoid burdening the DBMS with repeated identical queries. Do all the heavy big-volume processing where the data is, in the DBMS with procedures. You build your sawmill where the trees are. I know of one big application vendor whose payroll program is a straight port of an ISAM app to RDBMS. It takes 8 hours on big iron to 'balance the checkbook' of 40,000 employees. If this were done in procedures, it'd take minutes. I know of another big app vendor whose client interface has about 250 separate business methods. A typical user would do his/her work calling these methods about 50-100 times a day. More than 80% of these methods have processing that includes a fresh query to the DBMS to return a list of all the countries in Europe. I understand the political volatility in Europe has been higher recently... ;-)
Joe Weinstein at BEA
> -- > with regards > > Dusan Bolek Hans Forbrich - 25 Nov 2003 19:12 GMT > Hello, > [quoted text clipped - 8 lines] > world foundable, mistakes in development applications using Oracle > datatabase? Aside from the traditional onces (appropriate SGA sizing, BIND variables, etc.), some negative impact development philosophy/implementation items that I've observed:
1) Attempt to do the security in application - critical hole when multiple access techniques are allowed - requires duplication of the s/w - usually not as efficient as using Oracle's 'row level security'
1a) Not using Oracle's Resource manager 2) Attempt to do ALL application logic in front/mid tier - invariably moves excess amounts of data - move data should be restricted to minimum resolved set so use joins and where clause properly
3) Using a procedural language (C, C++, PL/SQL, VB) when the same results could be obtained by one (or more) well-thought-out SQL statement(s) - usually a procedural language involves stepping through a cursor - if a cursor loop ONLY involves data transformation, chances are high that it could be accomplished by a single SQL statement with appropriate selectivity
4) Using TEMP tables - There are valid reasons for temp tables; - However, several uses of temp tables are simply a workaround to "advanced" isolation level capabilities found in some other RDBMSs. Oracle generally does not require such a workaround as it ensures that the data in a transaction can not be seen outside the transaction's session.
5) IMO, every index should be reviewed every 6 months toi determine whether it's needed - Too often have I seen indexes created to meet 1-shot or limited use requirements and left around; - Remove unneeded or unnecessary (or duplicate) indexes!
6) Reinventing the wheel. - I've seen a lot of organizations literally spend millions of dollars duplicating built-in functionality of (intermedia's text search, workflow, workspace management/long transaction, advanced queueing, htp/htf, advanced grouping functions, spatial/location management, etc.); - This not only wastes development effort and costs extra maintenance, it dilutes the value of the product, making it appear more expensive than it is; - Developers should know and understand Oracle's capabilities; - Reinventing the wheel invariably violates 1) 2) and 3) above.
7) Insisting in yet another instance on that machine, 'cause each application should have it's own instance - likely can't be tuned - wastes license $ due to semaphore, shmem and CPU overhead for each instance
8) Insisting that the machine should host ALL of the database instance, the access tier, the report generator, etc. - likely can't be tuned - wastes license $ (eg: need 4 CPU to do 2 CPU's worth of database work)
9) Using RBO; not running appropriate stats - In 9i, CBO is quite acceptable AND will take into account the changes in environment.
10) Not using appropriate index types (eg: Function Based indexes)
11) Not allowing the DBA to actually administer the DB - Some organizations don't get apprpriate DBA skills or allow developers to override DBA suggestions
12) Not evaluating or using appropriate options - Partitioning can and does provide significant performance and scalability benefits and is relatively easy to cost justify if the organization has any sort of 'business case justification' process.
13) Blindly following rules of thumb without understanding them.
Daniel Morgan - 25 Nov 2003 21:43 GMT > Hello, > [quoted text clipped - 13 lines] > > Dusan Bolek Having read everyone else's list.
1. Developers that don't understand the implications of the multiversion concurrency model.
2. Developers that don't understand the implications of reads not blocking writes and writes not blocking reads.
3. DBAs that don't understand the implications of ORA-01555.
4. Those schooled in other RDBMS products that insist on treating Oracle as if were the other product.
5. Organizations that don't conduct code reviews.
6. Developers of testing prorams that can't read code and don't know how to identify potential weaknesses and test for them.
 Signature Daniel Morgan http://www.outreach.washington.edu/ext/certificates/oad/oad_crs.asp http://www.outreach.washington.edu/ext/certificates/aoa/aoa_crs.asp damorgan@x.washington.edu (replace 'x' with a 'u' to reply)
Daniel Roy - 25 Nov 2003 22:14 GMT Norman did a great job listing the database development sins, but I have 1 more: Doing everything that Siebel enforces its customers to do! For example: - Enforce the use of RBO instead of CBO! - Rebuild indexes periodically, in the hope that "it will help performance" - create gazillions of indices on an OLTP system (it's not common in Siebel that more than 100 indices are defined on a table) - Have no reference integrity at the database level! - Avoid at all costs to use MV's or views!!!! (they prefer instead to use giant SQL statements).
Daniel Roy Siebel Developer!
> Hello, > [quoted text clipped - 8 lines] > world foundable, mistakes in development applications using Oracle > datatabase? Anurag Varma - 25 Nov 2003 22:28 GMT > Norman did a great job listing the database development sins, but I > have 1 more: Doing everything that Siebel enforces its customers to [quoted text clipped - 10 lines] > Daniel Roy > Siebel Developer! --snip--
he he .. and you would wonder how Tom Siebel who was former group vice president for Oracle USA could be incharge of an application which breaks so many rules in the Oracle world.
:) Maybe getting back at Larry and his customers ... Anurag
mcstock - 25 Nov 2003 23:08 GMT | > Norman did a great job listing the database development sins, but I | > have 1 more: Doing everything that Siebel enforces its customers to [quoted text clipped - 19 lines] | | Anurag well, after having spent the better part of the last 15 years cleaning up after Oracle consultants and untraining bad habits taught by Oracle Education, my experience would say that maybe it's just a matter of consistency
(of course, as an xoraclite, my observations may be tainted ;-{)
--mcs
Daniel Morgan - 25 Nov 2003 23:42 GMT >>Norman did a great job listing the database development sins, but I >>have 1 more: Doing everything that Siebel enforces its customers to [quoted text clipped - 19 lines] > > Anurag If Larry doesn't get PeopleSoft he may turn his attentions elsewhere. Hmmmm!
 Signature Daniel Morgan http://www.outreach.washington.edu/ext/certificates/oad/oad_crs.asp http://www.outreach.washington.edu/ext/certificates/aoa/aoa_crs.asp damorgan@x.washington.edu (replace 'x' with a 'u' to reply)
Hans Forbrich - 25 Nov 2003 23:51 GMT > If Larry doesn't get PeopleSoft he may turn his attentions elsewhere. Hmmmm! I believe he's on record saying that Peoplesoft is the first of a set. Others have been identified as 'possible when the price is right'.
(IIRC, taken from an interview in Internet Week, May/June timeframe - roughly the same weekas the original bid for PeopleSoft.)
Joel Garry - 25 Nov 2003 23:48 GMT > Hello, > [quoted text clipped - 8 lines] > world foundable, mistakes in development applications using Oracle > datatabase? Letting developers into the production database.
jg -- @home.com is bogus. http://www.gripe2ed.com/scoop/story/2003/11/24/02356/143
Galen Boyer - 28 Nov 2003 00:48 GMT > Letting developers into the production database. Having DBA's make decisions about an application's database when they haven't been involved in the datamodeling or coding effort.
 Signature Galen Boyer
Hans Forbrich - 28 Nov 2003 09:12 GMT > > Letting developers into the production database. > > Having DBA's make decisions about an application's database when > they haven't been involved in the datamodeling or coding effort. Avoiding DBA (or at least data architect) input during design.
Having developers with no datamodeling experience dictate table and column definitions.
Galen Boyer - 28 Nov 2003 12:34 GMT >> > Letting developers into the production database. >> [quoted text clipped - 3 lines] > > Avoiding DBA (or at least data architect) input during design. That's the point. There are too many places where the DBA's aren't involved in anything but locking down the database. They shouldn't even be involved in that if that's all they do.
> Having developers with no datamodeling experience dictate table > and column definitions. Having DBA's that no nothing about the application development dictate coding policies.
 Signature Galen Boyer
Daniel Morgan - 28 Nov 2003 18:37 GMT >>>Letting developers into the production database. >> [quoted text clipped - 5 lines] > Having developers with no datamodeling experience dictate table and > column definitions. A number of you have hit this point from various angles and I would like to try to merge them all and thus throw gasoline on the fire.
1. There are DBAs that think their job consists of backup-recovery, creating users, and denying access to everything because end-users only create problems. I would cateogize them, for the most part, as DBAs that couldn't write and debug a stored procedure if you threatened them with a sharp object.
2. There are DBAs that are also competent developers having come up through the ranks and are just sick and tired of developers that throw junk over the cubicle wall, don't run explain plan, don't use bind variables, and haven't a clue of the implications of what they write, and whine like babies when the phrase "code review" is mentioned.
3. There are developers that are competent and can't understand why DBAs stand in their way trying to get the job done.
4. There are mnagers that don't have the skill set required to define projects, hire good people, fire bad people, and train those that need more skill to do their job properly.
Tackle number 4 and the other three resolve themselves quickly.
 Signature Daniel Morgan http://www.outreach.washington.edu/ext/certificates/oad/oad_crs.asp http://www.outreach.washington.edu/ext/certificates/aoa/aoa_crs.asp damorgan@x.washington.edu (replace 'x' with a 'u' to reply)
Jeff - 01 Dec 2003 14:32 GMT >2. There are DBAs that are also competent developers having come up >through the ranks and are just sick and tired of developers that throw >junk over the cubicle wall, don't run explain plan, don't use bind >variables, and haven't a clue of the implications of what they write, >and whine like babies when the phrase "code review" is mentioned. Do you write for Dilbert? :-)
>4. There are mnagers that don't have the skill set required to define >projects, hire good people, fire bad people, and train those that need >more skill to do their job properly. > >Tackle number 4 and the other three resolve themselves quickly. Abso-frikkin-lutely!
Jeff - 01 Dec 2003 14:28 GMT >> > Letting developers into the production database. >> [quoted text clipped - 5 lines] >Having developers with no datamodeling experience dictate table and >column definitions. Having developers with no datamodeling experience.
Jeff - 01 Dec 2003 14:27 GMT >> Letting developers into the production database. > >Having DBA's make decisions about an application's database when >they haven't been involved in the datamodeling or coding effort. LOL.
Ron Reidy - 26 Nov 2003 02:26 GMT 1. Developers that can't/won't read the docs. 2. Developers that can't/won't take advantage of Oracle strengths (black box phenomona). 3. Lack of bind vars; lack of reusable SQL; too much LIOs. 4. Overuse of hints w/o understanding the CBO and how it works. 5. Lack of budget to get the proper resources (people and hardware). 6. Propogation of "mythical cure-alls" for performance issues. 7. Badly architected software (see #3). 8. The "internet-time" syndrome vs. carefull, well thought out analysis.
Lot's more, I'm sure.
> Hello, > [quoted text clipped - 13 lines] > > Dusan Bolek
 Signature Ron Reidy Oracle DBA
Connor McDonald - 26 Nov 2003 12:23 GMT > Hello, > [quoted text clipped - 13 lines] > > Dusan Bolek Only one...
People on a project who aren't - passionate about what they do. - keen to learn
Everything else falls into place if you can satisfy the above
Daniel Morgan - 26 Nov 2003 17:08 GMT >>Hello, >> [quoted text clipped - 21 lines] > > Everything else falls into place if you can satisfy the above I'll add one more ...
Developers that seem incapable of understanding the basics of normalization and data integrity.
Again, today, I've been called into look at a database in which it is impossible to connect related events.
Example: 1. Cases have a case number 2. Citations have a case number and a citation number 3. Fees paid for citations have a case number and a payment number
If you can't see what's wrong ... you're part of the problem. ;-)
 Signature Daniel Morgan http://www.outreach.washington.edu/ext/certificates/oad/oad_crs.asp http://www.outreach.washington.edu/ext/certificates/aoa/aoa_crs.asp damorgan@x.washington.edu (replace 'x' with a 'u' to reply)
Billy Verreynne - 27 Nov 2003 06:42 GMT > Developers that seem incapable of understanding the basics of > normalization and data integrity. Amen to that.
> Again, today, I've been called into look at a database in which it is > impossible to connect related events. [quoted text clipped - 3 lines] > 2. Citations have a case number and a citation number > 3. Fees paid for citations have a case number and a payment number Almost the exact same problem here (dealing with billing issues and tariffs).
I'm still amazed at the "logic" employed to design relationships like that.
> If you can't see what's wrong ... you're part of the problem. ;-) Yeah.. and requires liberal application of the old lead pipe to correct the problem.
-- Billy
ctcgag@hotmail.com - 27 Nov 2003 20:10 GMT > Again, today, I've been called into look at a database in which it is > impossible to connect related events. [quoted text clipped - 5 lines] > > If you can't see what's wrong ... you're part of the problem. ;-) I'm not sure if I'm part of the problem or not. If the case-citation relationship is one-to-one but optional, I don't see the problem. If it's one to many, I do (unless you say that money is fungible and it doesn't matter what the payments are for as long as it adds up to the correct amount)
Xho
 Signature -------------------- http://NewsReader.Com/ -------------------- Usenet Newsgroup Service New Rate! $9.95/Month 50GB
Daniel Morgan - 28 Nov 2003 00:14 GMT >>Again, today, I've been called into look at a database in which it is >>impossible to connect related events. [quoted text clipped - 13 lines] > > Xho You may not be part of the problem but you are a strong candidate. ;-)
With the above you can link citations to cases. And you can link fees to cases. But you can not tell which fee goes with which citation.
The following simple change would provide referential integrity.
1. Cases have a case number 2. Citations have a case number and a citation number 3. Fees paid for citations have a citation number and a payment number.
The fees are linked to the case by means of the citation number providing a complete referential join.
This is a critical skill. And one I see all too rarely.
 Signature Daniel Morgan http://www.outreach.washington.edu/ext/certificates/oad/oad_crs.asp http://www.outreach.washington.edu/ext/certificates/aoa/aoa_crs.asp damorgan@x.washington.edu (replace 'x' with a 'u' to reply)
Jeff - 01 Dec 2003 14:13 GMT >This is a critical skill. And one I see all too rarely. That's very sad for you then, because that skill seems extremely fundamental--any database developer worth a shatt, should know that stuff backwards and forwards.
Mark D Powell - 26 Nov 2003 18:44 GMT > Hello, > [quoted text clipped - 8 lines] > world foundable, mistakes in development applications using Oracle > datatabase? Dusan, I would suggest you change your wording about issuing commit within a loop since in many cases the commits need to be performed within a loop. I think it is the frequency of commits rather than the logical coding structure that is the issue. In my opinion how frequently a commit should be issued is dependent on how likely another session will need update access to the rows being processed. In a transaction driven system the inventory rows may well require commits be issued on a per-row or very few row basis. While another program in the application, which is the only source of update activity for a table that has low select activity, would best be served by one or very few commits.
Be careful in the process of trying to get something into the hands of developers that you do not over simplify the problem and end up making a statement that you later consider "dumb". Writing for developers is not an easy task.
Good luck -- Mark D Powell --
Frank - 26 Nov 2003 21:38 GMT >>Hello, >> [quoted text clipped - 28 lines] > Good luck > -- Mark D Powell -- I'm sure Dusan referred to the "we have a possible 1555, so let's commit every ...uhm, 5000 processed rows". Committing within a loop to "prevent" 1555's from happening is creating the 1555's!
 Signature Regards, Frank van Bortel
ctcgag@hotmail.com - 30 Nov 2003 00:30 GMT > >>Hello, > >> [quoted text clipped - 31 lines] > I'm sure Dusan referred to the "we have a possible 1555, so let's > commit every ...uhm, 5000 processed rows". But I think this just demonstrates Mark's point. If you have to already know what he is talking about in order to know what he is talking about, then it isn't a very good educational document.
However, your supposition seems to require that developer are 1) reading the manuals and thinking carefully about what problems may arise, 2) taking pre-emptive action to forestall those problems, and 3) making things worse rather than better.
Now 3) I buy completely, but the first two seem foreign to most of my experience.
> Committing within a loop to "prevent" 1555's from happening is > creating the 1555's! I recall first encountering 1555s, and they were definately not created by committing inside a loop, because I wasn't doing that. In fact, the solution to the 1555s did involve moving the commit from outside the loop to inside a loop, but not doing it in the naive manner I first tried. (And it also solved the companion problem, of locking others out of the rows for too long).
So if you tell people that committing inside the loop is causing the 1555s, when those existed even before they moved the commit into the loop, it isn't going to inspire developers to take you seriously.
Xho
 Signature -------------------- http://NewsReader.Com/ -------------------- Usenet Newsgroup Service New Rate! $9.95/Month 50GB
Frank - 30 Nov 2003 16:47 GMT >>>>Hello, >>>> [quoted text clipped - 59 lines] > > Xho I did not say the opposite was true; having 1555's is not a sure sign of committing over loops (althought I tend to go look for them!!!), building them into a loop is a sure recipe for getting them - sooner or later.
 Signature Regards, Frank van Bortel
Daniel Morgan - 26 Nov 2003 21:55 GMT <snipped>
> Dusan, I would suggest you change your wording about issuing commit > within a loop since in many cases the commits need to be performed > within a loop. I think it is the frequency of commits rather than the > logical coding structure that is the issue. ><snipped> Going to take issue with you here Mark. Granted that nothing should ever been done dogmatically. But 98% of the time a commit in a loop is not needed, not desirable, the result of someone moving from another RDBMS or someone that has poorly configured rollback.
The rare times that a commit in a loop is required are the extreme exception.
 Signature Daniel Morgan http://www.outreach.washington.edu/ext/certificates/oad/oad_crs.asp http://www.outreach.washington.edu/ext/certificates/aoa/aoa_crs.asp damorgan@x.washington.edu (replace 'x' with a 'u' to reply)
Mark D Powell - 29 Nov 2003 22:29 GMT > <snipped> > > [quoted text clipped - 11 lines] > The rare times that a commit in a loop is required are the extreme > exception. Daniel, in the real world I would hazard to claim that most updates are done inside a loop because the majority of DML activity takes place in batch processing. It might be online batch or traditional batch but the programs have all have the same basic feature. The programs all perform their logic until there are no more input records whether those records come from a batch file or a driving cursor. In other words the programs all loop.
When the target rows are also the target of concurrent DML access then commits within a loop are a necessity because you cannot hold rows that are the target of OLTP activity for several seconds without adversely impacting performance. So the question is not whether you commit in a loop, but how frequently.
Snapshots too old are generally only a problem if the source and update target are the same table. When that is the case resorting to use of a driving table or similar technique will normally avoid the problem.
Most any shop that runs MRP II, or ERP, which is nearly every major corporation around has an environment that makes heavy use of batch syte processes. It may be a customer who triggers the job but the task processes the same logic for every target row in a loop. When you add the OLTP updates up and compare them to the number of rows processed via batch the majority of updates come from batch. And batch programs by nature involve a driving loop, hence my contention that it isn't doing updates in a loop structure that is the problem but rather the frequency of commit.
You can disagree, but I think your are failing to consider how much real world work is done in batch type processing.
-- Mark D Powell --
Daniel Morgan - 29 Nov 2003 23:53 GMT > You can disagree, but I think your are failing to consider how much > real world work is done in batch type processing. > > -- Mark D Powell -- And I will. I remember the first time I was a Boeing and wrote code that did that. Seemed reasonable at the time as I was bringing in gigabytes every weekend. But now with UNDO I'd argue just the opposite. Disk is so inexpensive it makes no sense not to just assign the equivalent of 1 or more drives, often 40+GB to UNDO and let it run to comletion.
But your point is valid in a datawarehouse situation. Can we at least agree that commits in OLTP loops is almost undoubtedly without merit.
 Signature Daniel Morgan http://www.outreach.washington.edu/ext/certificates/oad/oad_crs.asp http://www.outreach.washington.edu/ext/certificates/aoa/aoa_crs.asp damorgan@x.washington.edu (replace 'x' with a 'u' to reply)
Howard J. Rogers - 30 Nov 2003 02:46 GMT > > You can disagree, but I think your are failing to consider how much > > real world work is done in batch type processing. [quoted text clipped - 5 lines] > every weekend. But now with UNDO I'd argue just the opposite. Disk is so > inexpensive As one who frequently quotes the 'disk space is cheap' mantra, just for laughs, let me state that "proper" disk space is *not* cheap... EMC aren't exactly bargain basement material.
Furthermore, RAM is definitely not cheap, and the more disk space you have, the more blocks there are which need a home in the buffer cache.
So I'm not agreeing or disagreeing with you... just saying things aren't as simple as that, and there are subtleties to be considered.
>it makes no sense not to just assign the equivalent of 1 or > more drives, often 40+GB to UNDO and let it run to comletion. Bear in mind the Oracle automatic undo algorithm: the more space you assign to the undo tablespace, the more undo segments you end up acquiring. The more undo segments, the more undo segment header blocks. The more undo segment header blocks, the more Oracle overhead there is to manage the thing. And the more undo blocks in general, the bigger your buffer cache had better be.
The "sense" in automatic undo is that left to its own devices (ie, implemented poorly) it can consume resources at a rate of knots.
Regards HJR
> But your point is valid in a datawarehouse situation. Can we at least > agree that commits in OLTP loops is almost undoubtedly without merit. Daniel Morgan - 30 Nov 2003 18:42 GMT >>>You can disagree, but I think your are failing to consider how much >>>real world work is done in batch type processing. [quoted text clipped - 28 lines] > The "sense" in automatic undo is that left to its own devices (ie, > implemented poorly) it can consume resources at a rate of knots. A fascinating phrase. Please explain "rate of knots." It is new to me.
I'm not writing some new nonsense mythology that goes "NEVER commit in a loop." I'm meerly arguing that it should be done if required. And "if required" means you've tried it without and can't make it work. Lets remember that almost every post on UNDO, including yours, have advocated pushing the undo retention out as far as possible to enable using DBMS_FLASHBACK for recovery.
In the end ... the compromise to be made ... is between the cost of hardware and the cost of an ORA-01555. I'll continue to argue that the hardware is far less expensive in almost every situation.
 Signature Daniel Morgan http://www.outreach.washington.edu/ext/certificates/oad/oad_crs.asp http://www.outreach.washington.edu/ext/certificates/aoa/aoa_crs.asp damorgan@x.washington.edu (replace 'x' with a 'u' to reply)
Howard J. Rogers - 30 Nov 2003 19:16 GMT > > As one who frequently quotes the 'disk space is cheap' mantra, just for > > laughs, let me state that "proper" disk space is *not* cheap... EMC aren't [quoted text clipped - 20 lines] > > A fascinating phrase. Please explain "rate of knots." It is new to me. Speedily.
> I'm not writing some new nonsense mythology that goes "NEVER commit in a > loop." I'm meerly arguing that it should be done if required. And "if > required" means you've tried it without and can't make it work. Lets > remember that almost every post on UNDO, including yours, have advocated > pushing the undo retention out as far as possible to enable using > DBMS_FLASHBACK for recovery. That's actually not true. I did, for a time, recommend huge undo tablespaces precisely because I liked the idea of being able to flashback almost as far in the past as you could ever want to. But I quickly stopped suggesting that when Jonathan (I think) pointed out the impact of vast amounts of empty undo space on the 'allocate this new transaction to a new undo segment' algorithm. A bit of testing confirmed that I had just made my biggest-ever mistake in understanding Oracle. For the past year and more, I have therefore been saying that automatic undo needs to be set rather carefully, because otherwise you will swamp your buffer cache with Oracle overhead (ie undo segment headers for starters). Obviously, there is a tension between 'minimal but adequate' and 'I'd like to do flashback' (which is one reason why I think you'll find Jonathan doesn't like flashback very much). I actually like flashback a lot, and would arrange things accordingly... but like everything good in Oracle, it comes with a cost, and that cost needs to be monitored and tuned for.
> In the end ... the compromise to be made ... is between the cost of > hardware and the cost of an ORA-01555. I'll continue to argue that the > hardware is far less expensive in almost every situation. But ORA-1555 can be solved with a much more modest and realistic setting of undo_retention than you suggest I've been recommending in the past.... how long is your longest query (answerable from v$undostat) should govern undo_retention. Not "push it out as far as possible" or throw entire hard disks at it.
In short, over-doing the undo tablespace can limit scalability and performance because you can kiss goodbye to rather a lot of your buffer cache in the process. And I've been consistent in saying that for a very, very long time.
Regards HJR
Daniel Morgan - 30 Nov 2003 21:47 GMT Comments in-line.
>>>As one who frequently quotes the 'disk space is cheap' mantra, just for >>>laughs, let me state that "proper" disk space is *not* cheap... EMC [quoted text clipped - 60 lines] > like everything good in Oracle, it comes with a cost, and that cost needs to > be monitored and tuned for. I apparently missed the change. A year ago you say. Hmmm. Well I've been building most OLTP systems with 24 hour undo retention and have yet to find a real-world penalty. Thanks for the heads-up. I will keep my eyes open a bit more.
>>In the end ... the compromise to be made ... is between the cost of >>hardware and the cost of an ORA-01555. I'll continue to argue that the [quoted text clipped - 5 lines] > undo_retention. Not "push it out as far as possible" or throw entire hard > disks at it. My use of undo retention at 24 hours has been based on the backup cycle rather than specifically addressing ORA-01555 which usually requires not more than 30 minutes.
> In short, over-doing the undo tablespace can limit scalability and > performance because you can kiss goodbye to rather a lot of your buffer [quoted text clipped - 3 lines] > Regards > HJR Not doubting you in any manner I went back to the Google archives and found this:
=================================== From: Howard J. Rogers (howardjr2000@yahoo.com.au) Subject: Re: How to calculate the optimal size for undo segments View: Complete Thread (6 articles) Original Format Newsgroups: comp.databases.oracle.server Date: 2003-03-07 04:17:01 PST
Of course: you can't measure these things until you've already got an undo tablespace, so what you are supposed to do is to create a massive tablespace that you reckon will contain the worst case you can possible imagine, and let your users do their thing.... that populates v$undostat, and you take it from there, sizing the tablespace down or up as the case may be. ===================================
It was in that thread that you wrote:
"So there's a balance to be struck between oodles of undo space and a working buffer cache."
So being a Yank I went for "massive" as I didn't understand what an oodle is. Thanks for the clarification. ;-)
 Signature Daniel Morgan http://www.outreach.washington.edu/ext/certificates/oad/oad_crs.asp http://www.outreach.washington.edu/ext/certificates/aoa/aoa_crs.asp damorgan@x.washington.edu (replace 'x' with a 'u' to reply)
Mark D Powell - 30 Nov 2003 19:05 GMT > > You can disagree, but I think your are failing to consider how much > > real world work is done in batch type processing. [quoted text clipped - 9 lines] > But your point is valid in a datawarehouse situation. Can we at least > agree that commits in OLTP loops is almost undoubtedly without merit. I think we still disagree. In a data warehouse I would expect there to be few updaters, but in a large heavily used OLTP where concurrent access to the same rows is a reality then commits within a loop will be a necessity for some processes.
There are applications where this may not be necessary and low work loads allow less frequent commits, but I have yet to look at an MRP or ERP system that did not contain commits within a loop.
If you this is bad design then I will just refer you to Material Requirement Planning: The New Way of Life in Production and Inventory Management by Joseph Orlicky published by McGraw-Hill in 1975. When I sought certification by the American Production and Inventory Society, APICS, this was one of the recommended texts. It describes what MRP systems have to do. Seriously, if you can think of how to perform the work without using loops, and allowing concurrent update access without commits, then you can probably make a small fortune since I know a lot of firms that would be interested.
I think how someone views this argument might be colored by the types of applications they have experience with. I have a lot of experience with applications with high concurrent user work loads that target a few, large objects and concurrency is a priority. Smaller systems would have different characteristics.
The original point was that blanket statements are dangerous because any one rule may not be suitable for different applications and under varying work loads. While I agree there are definite drawbacks to committing after updating on a per row basis there are situations where doing so is a necessity and guidelines produced by the DBA for developers need to allow/account for these situations. Rather than say do this or do not do that the guidelines should lay out the side effects of design decisions based on how Oracle handles the work. Then it is up to the developers who should know the data better than the DBA to make the right choices.
IMHO -- Mark D Powell --
Daniel Morgan - 30 Nov 2003 21:34 GMT >>>You can disagree, but I think your are failing to consider how much >>>real world work is done in batch type processing. [quoted text clipped - 11 lines] > > I think we still disagree. I think we don't.
> The original point was that blanket statements are dangerous Which is why we don't. Because on this I agree as I've stated.
because
> any one rule may not be suitable for different applications and under > varying work loads. And we agree again.
While I agree there are definite drawbacks to
> committing after updating on a per row basis there are situations > where doing so is a necessity and guidelines produced by the DBA for > developers need to allow/account for these situations. And yet again.
> IMHO -- Mark D Powell -- My point, to repeat myself, was that one should try without commits inside of loops as an initial approach and only commit inside of a loop if as a back-up strategy. Too many times commits within a loop are part of the initial build.
Can we agree to agree? ;-)
 Signature Daniel Morgan http://www.outreach.washington.edu/ext/certificates/oad/oad_crs.asp http://www.outreach.washington.edu/ext/certificates/aoa/aoa_crs.asp damorgan@x.washington.edu (replace 'x' with a 'u' to reply)
Joel Garry - 01 Dec 2003 23:47 GMT > >>>You can disagree, but I think your are failing to consider how much > >>>real world work is done in batch type processing. [quoted text clipped - 39 lines] > > Can we agree to agree? ;-) I'm not sure you've agreed with Mark's statement, which statement implies that you should not try it first because you must take into account the overall functioning of the system, which is often less pure OLTP than one would assume from your repetition.
I've seen it over and over again, especially with Mark's examples of MRP and ERP systems, they just aren't pure. Worse, the data warehouse project that could perhaps partly purify them drags on unfinished until they get news that, for example, now we can have multiple block sizes, so why _not_ have everything in one giant instance for up-to-the-second data? :-O
Time for an Oracle Purity test! :-)
jg -- @home.com is bogus. Death of Moore's Law, news at 11 http://www.msnbc.com/news/999894.asp?cp1=1
Jeff - 01 Dec 2003 13:47 GMT >developers need to allow/account for these situations. Rather than >say do this or do not do that the guidelines should lay out the side [quoted text clipped - 3 lines] > >IMHO -- Mark D Powell -- Bravo. Don't treat developers like dumb-a.ses unless you want them to be dumb-a.ses.
Joel Garry - 01 Dec 2003 23:31 GMT > >developers need to allow/account for these situations. Rather than > >say do this or do not do that the guidelines should lay out the side [quoted text clipped - 6 lines] > Bravo. Don't treat developers like dumb-a.ses unless you want them to be > dumb-a.ses. So what do you do if they are dumb-a.ses before they've been treated?
:-) Seriously, some of the worst I've seen are newbies with a chip on their shoulder. They come at it with the attitude that the way they learned it somewhere else is the way it should be done, and attempting to lay out guidelines as Mark suggests is just flamebait for them. Billy V's Big Lead Pipe Treatment becomes unavoidable. The Dilbertian side effect is management decides that some overly-restrictive rules should be placed on everyone.
Personally, I'd rather that developers be encouraged to be creative, with a filtering process that explicates Oracle's issues with attempted solutions (in a practical sense that means Explain Plan and code reviews, as Daniel points out). This tends to be at odds with classical formal design processes.
Also, I would expect DBA's to be more familiar with global data issues than developers, in a large environment.
jg -- @home.com is bogus. http://www.signonsandiego.com/news/uniontrib/mon/business/news_mz1b1im.html
Jeff - 02 Dec 2003 13:42 GMT >So what do you do if they are dumb-a.ses before they've been treated? >:-) Re-examine your hiring and/or compensation practices. In other words, avoid hiring dumb-a.ses in the first place. ;-)
>Seriously, some of the worst I've seen are newbies with a chip on >their shoulder. They come at it with the attitude that the way they Newbies or wet-behind-the-ears young'uns fresh out of college that just need to shut up and mature a bit? :-)
>Personally, I'd rather that developers be encouraged to be creative, >with a filtering process that explicates Oracle's issues with >attempted solutions (in a practical sense that means Explain Plan and >code reviews, as Daniel points out). This tends to be at odds with >classical formal design processes. What're "classical formal design processes?" Because I think code reviews with an experienced DBA that provides developers with insight in how to do things better should be welcomed... as long as the DBA isn't full of shatt (or himself) and/or an a.s about it. I think experienced developers would, at least.
Joel Garry - 03 Dec 2003 18:42 GMT > >So what do you do if they are dumb-a.ses before they've been treated? > >:-) > > Re-examine your hiring and/or compensation practices. In other words, avoid > hiring dumb-a.ses in the first place. ;-) Guess you've never been a contractor in a gummint shop ;-)
> >Seriously, some of the worst I've seen are newbies with a chip on > >their shoulder. They come at it with the attitude that the way they > > Newbies or wet-behind-the-ears young'uns fresh out of college that just need > to shut up and mature a bit? :-) Both. Some of the worst have been customers hiring developers out of vendors. So they are new to the shop, have the ability to walk on water as far as the customer is concerned, and have a lot of experience at the vendor - which would be one of the same vendors that do some of the worst deadly sins, often in the name of database heterogeneity. And since they can walk on water, they can poke data and code into the production db... right?
> >Personally, I'd rather that developers be encouraged to be creative, > >with a filtering process that explicates Oracle's issues with [quoted text clipped - 7 lines] > himself) and/or an a.s about it. I think experienced developers would, at > least. By that I mean doing all design work and pseuodocode (or whatever the particular process favors) up front, and the developers just grind code. The diametric opposite is Rapid Application Development (RAD, or similar buzzwords like Extreme Programming). Personally I prefer a middle ground with a bias toward RAD, with care given to watch for the downsides of each. To me, the power to have excellent development lies in the feedback of an iterative cycle. The top-down approach coupled with deadlines tends to give code reviews short shrift. Total RAD can wind up with an unmaintainable mess, especially when customers can too easily modify vendor code.
Not that I'm complaining about any of this, it means full employment for me. I'm just repelled and fascinated by the horror stories.
jg -- @home.com is bogus. http://www.signonsandiego.com/news/uniontrib/tue/business/news_1b2peregrine.html
Mark D Powell - 03 Dec 2003 19:45 GMT > >So what do you do if they are dumb-a.ses before they've been treated? > >:-) [quoted text clipped - 19 lines] > himself) and/or an a.s about it. I think experienced developers would, at > least. The problem with code walk-throughs is that by the time they happen it is offen too late to change the design. Developers need to involve the DBA with a design walk-thru before code is written. In my experience the majority of the time the DBA does not see the code until there is a production performance problem. By then major design changes and different approaches are "too late".
Just another challenge -- Mark D Powell --
Jeff - 04 Dec 2003 14:01 GMT >The problem with code walk-throughs is that by the time they happen it >is offen too late to change the design. Developers need to involve >the DBA with a design walk-thru before code is written. In my >experience the majority of the time the DBA does not see the code >until there is a production performance problem. By then major design >changes and different approaches are "too late". I agree 100% that a DBA should be part of the design team along with developers. If they aren't... if DBA's are only brought in at production time or after the "design concrete" has set, I'd say that's a management problem, not a developer problem. Maybe I'm naive or we've different ideas of what a "developer" does.
Aside to Joel: I've never worked in a top-down environment... nor have I a lot of experience (none positive, that's for sure) with working with outside vendors/consultants. So, I'm rather speaking to permanent shops with permanent employees.
Daniel Morgan - 04 Dec 2003 17:39 GMT Comments in-line.
>>The problem with code walk-throughs is that by the time they happen it >>is offen too late to change the design. Developers need to involve [quoted text clipped - 8 lines] > not a developer problem. Maybe I'm naive or we've different ideas of what a > "developer" does. You are naive. There are many organizations, I think most, where the DBA is the last one to see the application.
> Aside to Joel: I've never worked in a top-down environment... nor have I a lot > of experience (none positive, that's for sure) with working with outside > vendors/consultants. So, I'm rather speaking to permanent shops with > permanent employees. The one thing that needs to be added to this discussion is the huge number of Oracle DBAs that think their job consists of install, patch, lock down, backup and recover. And most pray they never have to do the recover step because they've never done it before.
If DBAs want to be considered valuable to a development team they must:
1. Learn to read and write PL/SQL well enough to give advice. 2. Develop OO, Java, UML, and other skills to match development. 3. Keep their skills up to the current version. 4. Not treat developers like they are the enemy. There is not harm in giving developes select on v_$session or access to dbms_profiler.
Too often the DBA's attitude and skills (or the lack thereof) are the root reason the DBA is excluded.
 Signature Daniel Morgan http://www.outreach.washington.edu/ext/certificates/oad/oad_crs.asp http://www.outreach.washington.edu/ext/certificates/aoa/aoa_crs.asp damorgan@x.washington.edu (replace 'x' with a 'u' to reply)
Galen Boyer - 05 Dec 2003 04:18 GMT > Too often the DBA's attitude and skills (or the lack thereof) > are the root reason the DBA is excluded. Daniel,
I completely agree with you on this one. Most DBA departments (not the dba's themselves) are run as though the DBA departments are "blessing" the different groups with their godlike knowledge. Then, when I talk to the DBAs directly, we are all the same. We love the technology and want to make great applications and we are generally just completely interested in ALL phases of the project. To me, a DBA and a developer are both the same type of person just on different skill tracks.
 Signature Galen Boyer
Daniel Morgan - 05 Dec 2003 07:28 GMT >>Too often the DBA's attitude and skills (or the lack thereof) >>are the root reason the DBA is excluded. [quoted text clipped - 9 lines] > project. To me, a DBA and a developer are both the same type of > person just on different skill tracks. You are correct and you point out another problem. The division of DBAs and developers into different departments with different budgets and different managers. Can't think of a better way to NOT create a team atmosphere.
Remember my 4th point ... managers that can't or won't? This screams of the same issue. What person in upper management thinks it is a good idea to separate DBAs from developers? And why isn't that person the one being downsized out of his or her job? I mean any reason other than the issue of who plays golf with who?
 Signature Daniel Morgan http://www.outreach.washington.edu/ext/certificates/oad/oad_crs.asp http://www.outreach.washington.edu/ext/certificates/aoa/aoa_crs.asp damorgan@x.washington.edu (replace 'x' with a 'u' to reply)
Galen Boyer - 05 Dec 2003 15:57 GMT > The division of DBAs and developers into different departments > with different budgets and different managers. Can't think of a > better way to NOT create a team atmosphere. We have this at my place of work. The wall between my database team and the DBA team of our parent company was such that I wasn't given a lick of access to any of our development database machines. I asked for dba access to the development machines they wanted us to use and they got all up in arms about security and other production concerns. These machines were to be used by my team and my team only. They have never once asked my about my designs or layouts or anything. The only thing they wanted to make sure of was that we were going through their layers to get anything done. Here is my answer in a thread that I was involved in. The thread I was involved in is probably reconstructed by many here from similar pleas for sane viewpoints from the DBA crews.
To your question about dba access, I will answer from my personal viewpoint. (ie, I don't want to talk for our overall architect) . I don't understand how the database architect for the whole project isn't given full access to all databases designated for development purposes for the project. These are development instances. They shouldn't be locked down as production instances. The only thing that should be locked down is the way the app is interacting with the database, because we are building a production app. We need to be certain that we hamper the application in exactly the same way it will be hampered in production. But, we don't need to hamper the development process with the same restrictions. When a construction crew is building a new home, they don't put the locks on the doors and windows and make the construction crew knock on the door to get in to do their work, but they certainly make sure the home is put together, just like it will be when it is sold. Before it is put on the market and sold, the security against even the guys that built it is then, and only then, put in place.
 Signature Galen Boyer
Daniel Morgan - 05 Dec 2003 19:03 GMT >>The division of DBAs and developers into different departments >>with different budgets and different managers. Can't think of a [quoted text clipped - 33 lines] > put on the market and sold, the security against even the > guys that built it is then, and only then, put in place. So somehow the DBA team expects the developers to develop a good application without access to DBMS_PROFILER, without access to v$ magic views, and without access to other tables and views critical to understanding what is happening and optimizing performance and scalability. Good one.
Then I imagine they sit around drinking coffee and whining about the poor quality of apps developed by your team.
Once again the issue goes straight to management incompetence. The managers in charge of these teams are the root source of the problem. They are doing nothing to foster cooperation.
 Signature Daniel Morgan http://www.outreach.washington.edu/ext/certificates/oad/oad_crs.asp http://www.outreach.washington.edu/ext/certificates/aoa/aoa_crs.asp damorgan@x.washington.edu (replace 'x' with a 'u' to reply)
Galen Boyer - 05 Dec 2003 19:30 GMT > So somehow the DBA team expects the developers to develop a > good application without access to DBMS_PROFILER, without [quoted text clipped - 4 lines] > Then I imagine they sit around drinking coffee and whining > about the poor quality of apps developed by your team. Actually, that is about it. After many battles and executives learning how much my team was hampered in trying to get our app tested on our different deployment environments, I now have full control of now only the development instances, but also complete access to the production instances. (I still relegate the production DBA responsibilities to the DBA team)
The business folks paying the checks weren't happy to learn that bureucracy was in the way of progress. They told the dba team to just open up the doors, completely.
> Once again the issue goes straight to management > incompetence. The managers in charge of these teams are the > root source of the problem. They are doing nothing to foster > cooperation. Its really that my parent company has never had a group willing to fight for what was needed. Everybody was always located on sight in the motherland headquarters downtown New York. The subsidiary I work for is located in Boston. The people making these ridiculous decrees had to answer to a group of developers and architects that they didn't manage. When the questions started coming out and climbing up the email escalation threads, the business folks started waking up to the power that the internal IT group held, and they didn't like it, one bit.
 Signature Galen Boyer
JEDIDIAH - 06 Dec 2003 01:24 GMT >> So somehow the DBA team expects the developers to develop a >> good application without access to DBMS_PROFILER, without [quoted text clipped - 15 lines] > bureucracy was in the way of progress. They told the dba team to > just open up the doors, completely. I can't imagine the production DBA's were any too happy about this. The last shop I worked in that was subjected to such shenanigans found it's entire Oracle DBA and Unix Sysadmin staff flee in a mass exodus.
NO ONE wanted to held responsible for possible phantom screwups coming from the development side of the shop.
[deletia]
You traded one insanity for another.
 Signature There's no reason not to contribute to the stone soup if those ||| contributions are not critical to your competitive edge. / | \
Galen Boyer - 08 Dec 2003 03:39 GMT >> The business folks paying the checks weren't happy to learn >> that bureucracy was in the way of progress. They told the dba [quoted text clipped - 7 lines] > NO ONE wanted to held responsible for possible phantom screwups > coming from the development side of the shop. Well, if the DBA team were even remotely interested in being part of our development solution, if they wanted to do code reviews and analyze database designs and basically get involved in our development effort, then, well I'd have been all ears. I invited them into our CVS code base numerous times and they declined. Our code has been in source control since day one. Not one, let me count them for you, not one DBA looked at it. I installed an internal newserver where we discussed all our technical issues at a level you see here. I invited them into that arena, they declined. I offered to work on a database module in CVS with them so we could have a code base that all those doing database work in our company could use. They declined. I never, ever wanted to not have a dba crew around. And, you know what? I still have a dba crew around. They are in charge of our production instance. I have access to it, can do data uploads and patch our instances, but they take care of backup and recovery. With the level of their involvement in the development cycle, which was zilch, this is exactly what they should do, and this is exactly what they ONLY should do.
> You traded one insanity for another. You are making a judgement based on different circumstances or just based on the titles of the characters involved in the story.
 Signature Galen Boyer
Paul Drake - 06 Dec 2003 04:42 GMT > > The division of DBAs and developers into different departments > > with different budgets and different managers. Can't think of a [quoted text clipped - 33 lines] > put on the market and sold, the security against even the > guys that built it is then, and only then, put in place. what privileges do you actually require?
What if the DBA (person) removed all of the privileges from the DBA role except for create session, table, type, view, trigger, sequence, synonym and alter session?
the problem with granting an app_owner schema the role DBA is that then the application is coded depending upon the DBA role (and usually, all of the sys_privs that are in that role, that the account invariably grants itself directly). it is such a PITA to get changes made to remove queries that hit the dba_ views (such as dba_cons_columns for RI errors). If the developers can't code against the dba_% views, but are limited to the all_ views, you don't have as many issues when the code runs on a qa db where the app owner account does not have the DBA role granted to it.
If you are using a single app_owner schema, you don't need any of the ANY privs. Yes, you need dictionary access in development, so the developer role gets dictionary access.
You only think that you need the DBA role granted to you. You most likely do not. The more you read about security, the more that you'll agree with this viewpoint (IMHO).
This is a gross oversimplification, I'm sure that you can find exceptions, but I believe it to be generally true.
Give the developers enough privileges to create objects in the own schemas in dev to get their work done, but not enough sys_privs to alter the app_owner schema(s) themselves.
Pd
Daniel Morgan - 06 Dec 2003 08:05 GMT Comments in-line.
> what privileges do you actually require? Select on v$ magic views required to see what is actually happening. One can start with v_$reserve_words and v_$open_cursor and if you have any imagination you can come up with dozens more.
> What if the DBA (person) removed all of the privileges from the DBA > role except for create session, table, type, view, trigger, sequence, > synonym and alter session? If you read Oracle instructions, which apparently few do, no one should be granted the DBA role. The DBA role, like CONNECT and RESOURCE are intended in the same manner as SCOTT/TIGER. They are supposed to be dropped following installation and new roles created appropriate to employee job requirements. The rule is give employees the privileges they need to do their job. No privilege they don't need any more than denying privileges they do need.
> the problem with granting an app_owner schema the role DBA is that This is a red herring. The DBA role, as provided by Oracle is supposed to be dropped.
> The more you read about security, the more that you'll agree with this > viewpoint (IMHO). In development. Sorry I don't buy you argument. In test yes. In staging yes. In production absolutely. But if developers do stupid things in development their work fails in test. So what? And that is what code reviews are supposed to catch long before anything goes to test. What you appear to be doing is handcuffing everyone because someone might do something if you let them.
> This is a gross oversimplification, I'm sure that you can find > exceptions, but I believe it to be generally true. I agree that it is a gross oversimplification. But I do not agree that it is generally true. I think a more generally true statement would be that DBAs inhibit the ability of developers to do their job and claim it relates to security. Then whine about the poor quality of what developers develop when it finally gets to production.
> Give the developers enough privileges to create objects in the own > schemas in dev to get their work done, but not enough sys_privs to > alter the app_owner schema(s) themselves. > > Pd Why? Tell me exactly what it is the competently hired, competently trained, and competently managed developers are going to do if they actually modify the app_owner schema in development?
The only come up with two possible answers. 1. Produce better applications 2. Not hate DBAs
Both sound like good outcomes to me.
 Signature Daniel Morgan http://www.outreach.washington.edu/ext/certificates/oad/oad_crs.asp http://www.outreach.washington.edu/ext/certificates/aoa/aoa_crs.asp damorgan@x.washington.edu (replace 'x' with a 'u' to reply)
Galen Boyer - 06 Dec 2003 15:12 GMT > the problem with granting an app_owner schema the role DBA is > that then the application is coded depending upon the DBA role [quoted text clipped - 6 lines] > runs on a qa db where the app owner account does not have the > DBA role granted to it. You are missing my point. I never want the user that the application will log in as to have anything but the priviledges that will be granted to it in public. What I want is a user that has dba priviledges (or a form thereof) that can be used by me and the development crew for the sole purpose of modifying the database for developing the app. I, most definitely, want to hamper the application schema exactly as I plan to in production.
 Signature Galen Boyer
Daniel Morgan - 06 Dec 2003 15:59 GMT >>the problem with granting an app_owner schema the role DBA is >>that then the application is coded depending upon the DBA role [quoted text clipped - 14 lines] > database for developing the app. I, most definitely, want to > hamper the application schema exactly as I plan to in production. What Paul Drake and so many others miss is that the security policy of a company should be a written document. And the security rights and priviileges of an application should be part of a written specification.
If developers are required to create or modify applications based on those two documents then no application they create or modify can possibly violate the agreed security policies and specifications.
The only thing the developers could possibly do is make their jobs a bit easier. Little things like looking up reserved words, checking to make sure that all of their dynamic SQL is using bind variables, looking at the numbers of hard and soft parses: In short all of the things Tom Kyte does on his asktom website to demonstrate that things are working as they should.
The sad fact is, and since I don't know you Paul I'm not pointing at you so don't take this personally, the DBAs that are most paranoid about giving developers privileges in development environments, are the DBAs that know the least about development: The ones that can't actually write and debug PL/SQL.
 Signature Daniel Morgan http://www.outreach.washington.edu/ext/certificates/oad/oad_crs.asp http://www.outreach.washington.edu/ext/certificates/aoa/aoa_crs.asp damorgan@x.washington.edu (replace 'x' with a 'u' to reply)
JEDIDIAH - 07 Dec 2003 04:59 GMT [deletia]
> The sad fact is, and since I don't know you Paul I'm not pointing at you > so don't take this personally, the DBAs that are most paranoid about > giving developers privileges in development environments, are the DBAs > that know the least about development: The ones that can't actually > write and debug PL/SQL. No, it is far more likely that the paranoid DBA has little or no respect for the developers. Developer quality varies widely. Some can even be entrusted with the DBA role on production systems. However, most can't even be trusted to wipe their own noses.
It is safer to be paranoid.
"can't actually write and debug PL/SQL" is a common problem all round.
 Signature There's no reason not to contribute to the stone soup if those ||| contributions are not critical to your competitive edge. / | \
Daniel Morgan - 07 Dec 2003 07:02 GMT > [deletia] > [quoted text clipped - 12 lines] > > "can't actually write and debug PL/SQL" is a common problem all round. You've posted, Paul's posted, others have posted.
I have 10+ years working as an Oracle DBA and I am still looking for concrete, real-world, examples of problems caused in production by developers that were allowed to manage their own sandbox.
If there is something here other than paranoia ... name names. What actually happened? And if a properly written and approved design with code reviews would have solved the problem you don't have a point other than a lack of decent procedures.
I await those examples. I suspect many developers watching here are too.
 Signature Daniel Morgan http://www.outreach.washington.edu/ext/certificates/oad/oad_crs.asp http://www.outreach.washington.edu/ext/certificates/aoa/aoa_crs.asp damorgan@x.washington.edu (replace 'x' with a 'u' to reply)
Galen Boyer - 08 Dec 2003 03:40 GMT > No, it is far more likely that the paranoid DBA has little or > no respect for the developers. And somehow DBAs must have respect from developers? Why? Because they have the title of DBA?
> Developer quality varies widely. DBA quality varies widely.
 Signature Galen Boyer
Paul - 16 Dec 2003 00:32 GMT galenboyer@hotpop.com says...
> > No, it is far more likely that the paranoid DBA has little or > > no respect for the developers.
> And somehow DBAs must have respect from developers? Why? > Because they have the title of DBA? Some of them appear to think so!
> > Developer quality varies widely.
> DBA quality varies widely. Amen to that - as a developer who wants to make the switch to DBA, I can't think of anyone better than an ex-developer to know the faults that DBA's seem to develop - maybe I'll be cursing developers in the future, but AFAICS, what Daniel Morgan is saying is dead on - you give the developers the rope that they can use to hang themselves *_ON DEVELOPMENT SYSTEMS_*. That's the whole point of being a developer.
I also find the idea of a hermetic seal between developers and DBA's who are supposed to be working for the same company on the same product a bit difficult - maybe I'm better suited to smaller shops where I can be both simultaneously, or at least where there is an overlap of roles, and one side understands (or at least tries to) the other!
Paul...
 Signature plinehan x__AT__x yahoo x__DOT__x com
C++ Builder 5 SP1, Interbase 6.0.1.6 IBX 5.04 W2K Pro
Please do not top-post.
Niall Litchfield - 16 Dec 2003 08:56 GMT > I also find the idea of a hermetic seal between developers and DBA's who > are supposed to be working for the same company on the same product a > bit difficult - maybe I'm better suited to smaller shops where I can be > both simultaneously, or at least where there is an overlap of roles, and > one side understands (or at least tries to) the other! I think you are exactly correct here, though DBAs and Developers do tend to have somewhat different mindsets. I speculate that this comes from the pressure on developers to deliver 'solutions' extremely rapidly and to tight budgets whilst preferably using new technologies so as not to be obsolete. DBAs tend to be under pressure to provide available, secure, highly performing data stores.
 Signature Niall Litchfield Oracle DBA Audit Commission UK
Paul Drake - 06 Dec 2003 22:43 GMT > > the problem with granting an app_owner schema the role DBA is > > that then the application is coded depending upon the DBA role [quoted text clipped - 14 lines] > database for developing the app. I, most definitely, want to > hamper the application schema exactly as I plan to in production. and for that, you do not require the sys_privs that are granted by default to the dba role created by default in a new database.
you do not need the "ANY" sys_privs to do what you are describing above. if you have the create sys_privs granted directly or via a role, your can create anything that you can dream of under that app_owner schema.
its only when you reference objects outside of that schema that the "ANY" sys_privs are thought to be required.
Pd
Paul Drake - 07 Dec 2003 05:42 GMT > > the problem with granting an app_owner schema the role DBA is > > that then the application is coded depending upon the DBA role [quoted text clipped - 14 lines] > database for developing the app. I, most definitely, want to > hamper the application schema exactly as I plan to in production. oh. please allow me to take a step back for a second.
we now have 3 user types, so let us assume that roles are in use for simplicity.
you have an app_owner schema role. you have an application user role (unprivileged). you have a dba role (although not necessarily the role named "DBA" that is pre-built by dbca).
you want an account that has some sort of dba role that can alter database properties, and have access to the init.ora or spfile.
your existing dba is unavailable, overworked, not competent, uninterested or just plain has no clue and nothing to offer in terms of background, experience, etc.
I can see you wanting to be able to make changes to the database config so as to help optimize how the app code runs. you want the project to succeed.
I could also see where an existing dba (group) might not want you tweaking parameters without justification, this sounds not so much like a communal development environment, but a play area for you to hone your dba skills. personal workstation and laptops are a great place to experiment. If you have to support a horde of developers with this dev db, it might not be the best place for you to develop/hone your dba skills.
one possible approach is to get say 10% of your time allocated to the dba group, and gain their respect, work with them to optimize your development environment.
another possible approach is to completely break from having any involvement from your dbas employed by your company. you will likely lose out on assistance that they could provide.
most likely, you do not want a development crew making database changes en masse. if anything, give them alter session so that they can muck with things as far as their session is concerned, but not muck with other user's sessions, or instance-wide settings.
I'd prefer if you not use the term "public" but substitute the term "production". to me, "PUBLIC" is something that is granted (improperly?) where security is of absolutely no concern - dual, perhaps - where anyone that has a session, has read/execute on the object - or worse (think dba_tab_privs).
I am honestly trying to be constructive here, and might have as much to gain from these thought experiments as you.
Pd
Daniel Morgan - 07 Dec 2003 07:15 GMT Comments in-line.
> you want an account that has some sort of dba role that can alter > database properties, and have access to the init.ora or spfile. [quoted text clipped - 14 lines] > this dev db, it might not be the best place for you to develop/hone > your dba skills. If I may jump in for Galen here, he's probably smarter than me and in bed ... what you've written is not the point at all. But your attitude is exactly what causes developers so much grief.
Developers don't have machines at home that are like those at work. I don't know a single developer who has a personal HP9000 (well I have two but that's not the point a do mostly DBA work and teaching). Nor are there many with Sun 450s or machines running AIX or OS/390, etc. So lets not pretend one can learn on their personal PC that which will best optimize an application in a business environment.
> one possible approach is to get say 10% of your time allocated to the > dba group, and gain their respect, work with them to optimize your > development environment. Another possible approach would be for one of you that believes in locking down a development environment to put forth actual, factual examples of things that have gone wrong. And I don't mean things that potentially could or might go wrong. And I don't mean urban myths you heard about at a Christmas party. I mean things that actually have happened to you.
Much like the detectives on Dragnet ... I want the facts. ;-)
> another possible approach is to completely break from having any > involvement from your dbas employed by your company. you will likely > lose out on assistance that they could provide. Funny cooperation isn't one of your possible approaches. Why? My suggested approach, and the one I use is to ask for justification and grant anything and everything that will not cause a problem when the code movest to test. If they mess it up ... that is their right. Interestingly though ... it has been more than three years since I've seen a development box blow up because of what developers did. And the last one that did was when a junior member of the DBA team tried to show off at a level above her abilities.
> most likely, you do not want a development crew making database > changes en masse. And if they have management why would they?
> if anything, give them alter session so that they can muck with things > as far as their session is concerned, but not muck with other user's > sessions, or instance-wide settings. Why? Provide one actual example of something that actually happened and how this caused a problem.
> I am honestly trying to be constructive here, and might have as much > to gain from these thought experiments as you. > > Pd I'm sure you are. But the divide between you and me (and I am mostly a DBA) is as wide as the Grand Canyon. Between you and most deveopers the size of the Pacific Ocean.
I understand your concern about people doing stupid things. I have the same concerns. But I am still asking for and awaiting actual examples of things that have gone wrong. From where I sit you have crossed the line from concern into paranoia.
And not once has anyone addressed my statements about granting access to developers to the v$ magic views and many of the DBMS and UTL packages and I would bet a Latte' at Starbucks you don't give your developers access to them either.
 Signature Daniel Morgan http://www.outreach.washington.edu/ext/certificates/oad/oad_crs.asp http://www.outreach.washington.edu/ext/certificates/aoa/aoa_crs.asp damorgan@x.washington.edu (replace 'x' with a 'u' to reply)
Billy Verreynne - 08 Dec 2003 05:44 GMT Daniel Morgan <damorgan@x.washington.edu> wrote
> And not once has anyone addressed my statements about granting access to > developers to the v$ magic views and many of the DBMS and UTL packages > and I would bet a Latte' at Starbucks you don't give your developers > access to them either. Well, on our DEV platform I have given the developer accounts select access to the v$ magic views in a vain hope that with a little bit of self empowerment, they will get to know Oracle better and even use it better.
But then I still get stuff like this in production: begin insert statement (trigger updates the seq)
select max(seq) into :s from table; return( :s ) end;
Code review? Not here. Every developer writes perfect code. Why the hell get the DBA involved at all (except when you want to use him as a friggen reference manual when too lazy to look up the syntax for an Oracle function or SQL statement).
-- Billy
Jeff - 08 Dec 2003 13:47 GMT >Daniel Morgan <damorgan@x.washington.edu> wrote > [quoted text clipped - 16 lines] > return( :s ) >end; This isn't evidence of developers that aren't Oracle literate. This is evidence of under-competent (sic) developers. Code reviews can be mandated by management--and if a DBA brought this kind of code to a competent IT manager, he/she should certainly see the need for them.
Daniel Morgan - 08 Dec 2003 17:24 GMT > Daniel Morgan <damorgan@x.washington.edu> wrote > [quoted text clipped - 24 lines] > -- > Billy That's half a latte'. How about DBMS_PROFILER and other built-in packages?
If you get it right send me your postal address off-line and the latte' is yours.
 Signature Daniel Morgan http://www.outreach.washington.edu/ext/certificates/oad/oad_crs.asp http://www.outreach.washington.edu/ext/certificates/aoa/aoa_crs.asp damorgan@x.washington.edu (replace 'x' with a 'u' to reply)
Billy Verreynne - 09 Dec 2003 05:42 GMT Daniel Morgan <damorgan@x.washington.edu> wrote
> That's half a latte'. How about DBMS_PROFILER and other built-in packages? They don't know how to use expain plan. I have tried numerous times to get them to use TKPROF. Access to DBMS_PROFILER? They won't know what to do with it.
FWIW, they do have access to workspaces, replication, directory objects and a bunch of other stuff that I as an Oracle developer would love to have my hands on. Also trying to get them to use OEM and JDeveloper. But the only thing they can use is TOAD.... <sigh>
> If you get it right send me your postal address off-line and the latte' > is yours. Does it come with several cases of beer? ;-)
-- Billy
Daniel Morgan - 09 Dec 2003 07:09 GMT > Daniel Morgan <damorgan@x.washington.edu> wrote > >>That's half a latte'. How about DBMS_PROFILER and other built-in packages? > > They don't know how to use expain plan. Shame shame shame!
Better than what I'm really thinking.
I have tried numerous times to
> get them to use TKPROF. Access to DBMS_PROFILER? They won't know what > to do with it. I have a rule ... no code goes into production without accompanying explain plans and DBMS_PROFILER runs. It is part of the rules of engagement and should be mandatory everywhere.
Any developer can learn Explain Plan in five minutes: DBMS_PROFILER in thirty or less.
>>If you get it right send me your postal address off-line and the latte' >>is yours. [quoted text clipped - 3 lines] > -- > Billy I keep Chimay in the frige but it doesn't ship well.
 Signature Daniel Morgan http://www.outreach.washington.edu/ext/certificates/oad/oad_crs.asp http://www.outreach.washington.edu/ext/certificates/aoa/aoa_crs.asp damorgan@x.washington.edu (replace 'x' with a 'u' to reply)
Paul Drake - 09 Dec 2003 16:39 GMT > > Daniel Morgan <damorgan@x.washington.edu> wrote > > [quoted text clipped - 26 lines] > > I keep Chimay in the frige but it doesn't ship well. That, is a difficult task, as it keeps disappearing. East India Pale Ales ship very well, like from the UK, to Mumbai, across India.
I checked out your article on DBMS_PROFILER last night. NYOUG has an excellent technical journal, and is always looking for authors to submit papers/articles (http://www.nyoug.org) if you're interested.
btw - the link to the Oracle Trace Analyzer utility is obsolete. there is a download link in the Metalink note 224270.1 http://metalink.oracle.com
hth.
Pd
JEDIDIAH - 11 Dec 2003 16:03 GMT > Comments in-line. > [quoted text clipped - 27 lines] > not pretend one can learn on their personal PC that which will best > optimize an application in a business environment. What makes you think that a developer will necessarily have reasonable development hardware even at the office? Any place that I have ever worked (or ever supported) has had a shortage of suitable development machines.
Quite often, "someone's desktop" actually ends up being a dev or QA server.
I've seen this in environments where the corp was already spending millions on Oracle licenses and server hardware.
[deletia]
 Signature There's no reason not to contribute to the stone soup if those ||| contributions are not critical to your competitive edge. / | \
Galen Boyer - 08 Dec 2003 03:10 GMT > I am honestly trying to be constructive here, If only I believed you.
 Signature Galen Boyer
Joel Garry - 05 Dec 2003 19:27 GMT > Comments in-line. > [quoted text clipped - 23 lines] > lock down, backup and recover. And most pray they never have to do the > recover step because they've never done it before. Don't forget "tune system for high hit ratios and spend weekends making sure each table fits in one big initial extent." :-)
> If DBAs want to be considered valuable to a development team they must: > [quoted text clipped - 6 lines] > Too often the DBA's attitude and skills (or the lack thereof) are the > root reason the DBA is excluded. You are quite right, but the downside is spreading the skill set too thin. For many environments, I think there should be quite different production and development DBA's, the latter being more directly skilled in your list, perhaps even being titled "senior programmer." Hmmm, that came out different from what I started out to post...
jg -- @home.com is bogus. "Someone once said 'Less is More.' That person was an idiot. You don't ask your boss for less money or less vacation time." - radio ad.
Joel Garry - 05 Dec 2003 01:22 GMT > >The problem with code walk-throughs is that by the time they happen it > >is offen too late to change the design. Developers need to involve [quoted text clipped - 13 lines] > vendors/consultants. So, I'm rather speaking to permanent shops with > permanent employees. The particular environment I was referring to was a very large permanent shop with permanent employees. They had bought an off the shelf package and had a large effort to modify it to their purposes, both with contractors and permanent employees. In fact, there were two large shops, in competition to do the same thing. One shop was using a layered approach, putting all new objects in separate schemata with private synonyms, while the other hacked away. They both were using Oracle tools to reverse-engineer the product, and both were traditional top-down shops, newly coming to Oracle from COBOL, and neither had the type of dba that would be necessary for what we are discussing here. Eventually one shops methodology took over, due to political reasons. Somehow, it all got done, albeit slowly. Which is a lot more than can be said for their DW project.
"Management problem" is an understatement.
jg -- @home.com is bogus. http://www.sumeiyu.com/
Jeff - 05 Dec 2003 21:52 GMT >The particular environment I was referring to was a very large >permanent shop with permanent employees. They had bought an off the [quoted text clipped - 11 lines] > >"Management problem" is an understatement. You are describing my personal nightmare... one that I've fought tooth and nail against from the bottom up. I believe we could design and develop a brand new system that'd be as good or better than what you'd end up with by buying and extensively modifying something. My personal bias.
Daniel Morgan - 05 Dec 2003 22:55 GMT >>The particular environment I was referring to was a very large >>permanent shop with permanent employees. They had bought an off the [quoted text clipped - 16 lines] > brand new system that'd be as good or better than what you'd end up with by > buying and extensively modifying something. My personal bias. From my experience you'd be right far more than wrong. I am aware of a system in King County Washington, based on Informix. They had a quote to build from scratch for $700,000. They bought an OTS prouct for $70,000. They have now spent 2.1 million tax payer dollars trying to modify the beast.
Oh yeah. And no one likes it either.
 Signature Daniel Morgan http://www.outreach.washington.edu/ext/certificates/oad/oad_crs.asp http://www.outreach.washington.edu/ext/certificates/aoa/aoa_crs.asp damorgan@x.washington.edu (replace 'x' with a 'u' to reply)
Paul - 16 Dec 2003 00:40 GMT damorgan@x.washington.edu says...
> From my experience you'd be right far more than wrong. I am aware of a > system in King County Washington, based on Informix. They had a quote to > build from scratch for $700,000. They bought an OTS prouct for $70,000. > They have now spent 2.1 million tax payer dollars trying to modify the > beast. Where would we get work but for foo-bars like that?
Paul...
 Signature plinehan x__AT__x yahoo x__DOT__x com
C++ Builder 5 SP1, Interbase 6.0.1.6 IBX 5.04 W2K Pro
Please do not top-post.
Paul - 16 Dec 2003 00:18 GMT Mark.Powell@eds.com says...
> Then it is up to the developers who should know the data better than > the DBA to make the right choices. Are you suggesting that a DBA shouldn't have a notion about what data is in his databases, and not know why it is there and for what purpose it is being used?
OK, maybe you didn't say "not have a notion" but you definitely said "should know the data better than the DBA".
I disagree - a DBA should know more than application programmers about what's in the DB's he manages and why - how else can he make recommendations about how "things" could be done far better on a far far higher level than, perhaps, the programmers see the data? "Things" meaning "System organisation", "optimisation", &c. &c.
Paul...
> IMHO -- Mark D Powell --
 Signature plinehan x__AT__x yahoo x__DOT__x com
C++ Builder 5 SP1, Interbase 6.0.1.6 IBX 5.04 W2K Pro
Please do not top-post.
Galen Boyer - 16 Dec 2003 15:36 GMT > Mark.Powell@eds.com says... > [quoted text clipped - 14 lines] > programmers see the data? "Things" meaning "System > organisation", "optimisation", &c. &c. I'm not sure how a dba is supposed to do this. If the DBA is also the data architect, then, maybe. But then, the DBA will spend all his time exploring the business instead of architecting the database. I think it is a team effort. The DBA should be able to depend on a data architect to give him the needed information so he can make the decisions his application's datastore needs.
The developers are going to be the experts at what the sqlcode is doing and the DBA should be able to depend on them so he can help them right more performant code and also stay away from bad code.
I guess, nobody should think of themselves as the "sole" source of info.
 Signature Galen Boyer
Niall Litchfield - 16 Dec 2003 16:23 GMT > > Mark.Powell@eds.com says... > > [quoted text clipped - 26 lines] > doing and the DBA should be able to depend on them so he can help > them right more performant code and also stay away from bad code. writing code that peforms acceptably means knowing things like (assume OLTP for the moment)
How many transactions/day do we do? When do we do them? What is a transaction anyway? How many customers do we have? And products. How many order items would an order have on average - and what sort of range of values might items/order take. What batch processes are run against this data set. How long do they have to run. as well as how to write SQL\PL/SQL\JAVA\ASP
now you might call this design and have it as a separate role but my hunch is that DBAs tend to know these sorts of things far more than developers. Of course if you actually *sell* your product then you pretty much have no way of answering these questions, and then folk worry about software quality of 3rd party apps.
> I guess, nobody should think of themselves as the "sole" source > of info. I agree entirely.
 Signature Niall Litchfield Oracle DBA Audit Commission UK
Joel Garry - 27 Nov 2003 00:44 GMT > > Hello, > > [quoted text clipped - 20 lines] > activity for a table that has low select activity, would best be > served by one or very few commits. The problem here is beyond semantics. You are likely breaking up logical transactions into several physical transactions, which means you must go beyond the inherent Oracle mechanisms to maintain atomicity. Some people would be horrified. As someone who deals with your example every day, I say you are exactly right, and wonder why "transaction levels" have never been implemented into the mechanisms (the stuff [4GL/OCI generator] I work with actually had that documented as a future enhancement, but the future never seems to have arrived; there is a language construct for controlling how often a set of rows will commit, so it is easy to fix per loop/per site without major code hassles).
With what we have in Oracle now, there is a legitimate trade-off between your example and "correct" transactional programming. What Dusan needs is a rule that codifies that. There have certainly been some amazingly bad examples of loopy commit.
> Be careful in the process of trying to get something into the hands of > developers that you do not over simplify the problem and end up making > a statement that you later consider "dumb". Writing for developers is > not an easy task. You got that right! Beyond that, it is difficult to make such a list without sounding like, shall we say, authors who have been pummelled on this group for making lists.
Dusan, I encourage you to do this, and volunteer to edit for good English if you email me (s/home.com/cox.net). Apologies in advance if I'm a bit slow reading email.
jg -- @home.com is bogus. "The cursor..For loop approach for the update was calculated to take 53.7 years to complete!" - Ask Tom posting.
Alkos - 27 Nov 2003 09:23 GMT > Hello, > [quoted text clipped - 13 lines] > > Dusan Bolek Datatypes is an issue I'm constantly dealing with :-((
Using a unproper (is that correct English ?) datatype for column e.g. NUMBER for a date (number of days/seconds elapsed) VARCHAR2 instead of NUMBER for a numeric
Being unaware of nearly deprecated datatypes and keeping on using them (VARCHAR, RAW)
Oversizing variable length datatypes
hth Alkos
Sybrand Bakker - 28 Nov 2003 23:11 GMT >Datatypes is an issue I'm constantly dealing with :-(( > [quoted text clipped - 4 lines] >Being unaware of nearly deprecated datatypes and keeping on using them >(VARCHAR, RAW) Let's shake hands. I have to deal with an in-house developed app (yuck) in which all number(10) are stored as varchar2s, left-padded. Everytime when I set up a query filter in Toad I have to take those dratted leading 0s into account. I will remain silent about the performance impact.
-- Sybrand Bakker, Senior Oracle DBA
Alkos - 01 Dec 2003 08:24 GMT > >Datatypes is an issue I'm constantly dealing with :-(( > > [quoted text clipped - 13 lines] > -- > Sybrand Bakker, Senior Oracle DBA Hi Sybrand,
I think there is a conspiracy/contest between developers to make the use of datatypes to most obfuscated possible.
A recent example (simple one but hidden in a sql script of 1000 lines) Select bla bla from table1,table2 where join predicates and table1.status!=1;
=> ORA 1722 (status was VARCHAR2 but now contains only numeric)
Cheers, Alkos
Alkos - 01 Dec 2003 14:11 GMT > >Datatypes is an issue I'm constantly dealing with :-(( > > [quoted text clipped - 13 lines] > -- > Sybrand Bakker, Senior Oracle DBA Hi again,
Let me ask you something . . . . . . how does this app perform basic calculations (+,-,* or even /) ?
Alkos
Marc Blum - 27 Nov 2003 21:16 GMT (1) SELECT DISTINCT ...
(2) ... WHERE <attribute> IN (SELECT DISTINCT ...)
(3) LOOP EXECUTE IMMEDIATE some_sql; END LOOP;
(4) SELECT MAX(id) + 1 AS new_id FROM big_table
(5) WHERE TRUNC(date_attr) = TRUNC(date_variable)
-- Marc Blum mailto:blumNOSPAM@marcblum.de http://www.marcblum.de
Cris Carampa - 28 Nov 2003 08:36 GMT > (1) > SELECT DISTINCT ... Sorry for the naive question: what's wrong with SELECT DISTINCT?
Kind regards,
 Signature Cris Carampa (spamto:cris119@operamail.com)
"Poveri fanatici comunisti, noglobal e affetti dalla sindrome anti-microsoft" (gli utenti Linux secondo un poster di ICOD)
Alkos - 28 Nov 2003 09:43 GMT > > (1) > > SELECT DISTINCT ... [quoted text clipped - 8 lines] > "Poveri fanatici comunisti, noglobal e affetti dalla sindrome > anti-microsoft" (gli utenti Linux secondo un poster di ICOD) Most of the time, people use SELECT DISTINCT to get unique rows because they don't want to "bother" with the datamodel. They are lazy or hurried so they don't care about possible keys, about WHERE clauses selectivity and all that kind of things so quick and dirty solution to be sure to get unique rows : SELECT DISTINCT which implies a sort therefore more activity to get the same result as they may have gotten quicker with a little bit work on the WHERE clause.
Alkos
Cris Carampa - 28 Nov 2003 11:05 GMT > Most of the time, people use SELECT DISTINCT to get unique rows because > they don't want to "bother" with the datamodel. They are lazy or hurried so [quoted text clipped - 3 lines] > implies a sort therefore more activity to get the same result as they may > have gotten quicker with a little bit work on the WHERE clause. I do use SELECT DISTINCT when I have just to pick up distinct values from a table, and I don't need to count them or performing group functions amongs columns in the table. I assume the performance of this statement:
SELECT DISTINCT FOOCOL FROM FOOTAB ;
is better than the performance of this one:
SELECT FOOCOL, COUNT(FOOCOL) FROM FOOTAB GROUP BY FOOCOL ;
Or am I wrong?
 Signature Cris Carampa (spamto:cris119@operamail.com)
"Poveri fanatici comunisti, noglobal e affetti dalla sindrome anti-microsoft" (gli utenti Linux secondo un poster di ICOD)
Frank - 28 Nov 2003 11:46 GMT >> Most of the time, people use SELECT DISTINCT to get unique rows because >> they don't want to "bother" with the datamodel. They are lazy or [quoted text clipped - 18 lines] > > Or am I wrong? Consider
select distinct e.first_name from employees e, departments d;
versus
select e.first_name from employees e, departments d where e.department_id=d.department_id;
Oversimplified, of course, but it's the idea.
 Signature Regards, Frank van Bortel
Daniel Morgan - 28 Nov 2003 18:41 GMT >>> Most of the time, people use SELECT DISTINCT to get unique rows because >>> they don't want to "bother" with the datamodel. They are lazy or [quoted text clipped - 32 lines] > > Oversimplified, of course, but it's the idea. I see no relationship between your two queries.
The first one is a Cartesian join returning distinct first names. The second is an inner join that returns all first names. They do completely different things.
 Signature Daniel Morgan http://www.outreach.washington.edu/ext/certificates/oad/oad_crs.asp http://www.outreach.washington.edu/ext/certificates/aoa/aoa_crs.asp damorgan@x.washington.edu (replace 'x' with a 'u' to reply)
Jeff - 01 Dec 2003 14:22 GMT >I see no relationship between your two queries. Neither do I. He must have some real "winners" working with him if that's the difference between a "right" and "wrong" query. ;-)
Alkos - 01 Dec 2003 09:07 GMT > > Most of the time, people use SELECT DISTINCT to get unique rows because > > they don't want to "bother" with the datamodel. They are lazy or hurried so [quoted text clipped - 22 lines] > "Poveri fanatici comunisti, noglobal e affetti dalla sindrome > anti-microsoft" (gli utenti Linux secondo un poster di ICOD) Neither right nor wrong. Those 2 queries are different in purpose.
select distinct should only be used when a natural key doesn't exist to get unique rows. This situation is *always* the consequence of a poor datamodel design.
I use personnaly select distinct when i want to test the cardinality of a column : select count(*), count(distinct my col) from bla bla
HTH Alkos.
Marc Blum - 28 Nov 2003 16:40 GMT >> (1) >> SELECT DISTINCT ... > >Sorry for the naive question: what's wrong with SELECT DISTINCT? > >Kind regards, DISTINCT requieres a sort. Unnecessary sorts are main reason fpr performance problems.
In 100% of the cases, where I stumbled about a SELECT DISTINCT, it was coded for one of these three reasons:
(1) the developer didn't understand the datamodel
(2) the developer didn't understand the requierements he was implementing
(3) the datamodel was inappropriate to the requierements
Over the last three or four years, no single DISTINCT survided, after I encountered it.
-- Marc Blum mailto:blumNOSPAM@marcblum.de http://www.marcblum.de
Daniel Morgan - 28 Nov 2003 18:21 GMT >> (1) SELECT DISTINCT ... > > Sorry for the naive question: what's wrong with SELECT DISTINCT? > > Kind regards, Nothing when used appropriately. It has a cost associated with it and thus should only be used when it is required to produce the correct result.
 Signature Daniel Morgan http://www.outreach.washington.edu/ext/certificates/oad/oad_crs.asp http://www.outreach.washington.edu/ext/certificates/aoa/aoa_crs.asp damorgan@x.washington.edu (replace 'x' with a 'u' to reply)
Alkos - 28 Nov 2003 09:52 GMT > (1) > SELECT DISTINCT ... [quoted text clipped - 17 lines] > mailto:blumNOSPAM@marcblum.de > http://www.marcblum.de Hello,
Why (3) LOOP EXECUTE IMMEDIATE some_sql; END LOOP; should be avoided ??
OK, it's Dynamical SQL but it could be useful to write scalable batch scripts. Tell me if i'm wrong.
Cheers, Alkos
Holger Baer - 28 Nov 2003 11:00 GMT > Hello, > [quoted text clipped - 11 lines] > Cheers, > Alkos If you want it truely scalable, then you would do something like this (Pseudo code following, so don't trash me for inaccuracies or not existing function/procedures - it's just supposed to give the general idea)
dbms_sql.open_cursor dbms_sql.parse LOOP dbms_sql.bind dbms_sql.execute END LOOP dbms_sql.close_cursor
This way, you avoid soft parses (and why you should avoid them is something that you will know if you hang around Tom Kytes website for a few days at http://asktom.oracle.com )
Personally, I use execute immediate only for administrative tasks. But then, I'm most of the time the DBA ;-)
Cheers,
Holger
Alkos - 28 Nov 2003 11:07 GMT > > Hello, > > [quoted text clipped - 35 lines] > > Holger Hi Holger,
Ok, the advice should be : "don't use execute immediate within loops to execute DML queries"* what i understood first. I was wondering whether there was any hindrance using exec immdt for DDL in some cases. According to me, there is not example : open a cursor to list all tables owned by the current user. loop throuh the cursor exec immdt to grant another user with s,i,u,d on this table end loop (this is really pseudo-code,isn't it ? ;-)
Alkos
Holger Baer - 28 Nov 2003 13:03 GMT >>If you want it truely scalable, then you would do something like this >>(Pseudo code following, so don't trash me for inaccuracies or not [quoted text clipped - 24 lines] > Ok, the advice should be : > "don't use execute immediate within loops to execute DML queries"* That is exactly right.
> what i understood first. I was wondering whether there was any hindrance > using exec immdt for DDL in some cases. According to me, there is not [quoted text clipped - 4 lines] > end loop > (this is really pseudo-code,isn't it ? ;-) As pseudo as can be, eh? ;-)
But you're right, that's the way to use execute immediate, at least, that's what I use it for.
Cheers
Holger
|
|
|