[Scribus] Re: Ideas on collaboration

[Asif Lodhi]

Hi Craig,

On 6/25/05, Craig Ringer <craig at postnewspapers.com.au> wrote:
> On Sat, 2005-06-25 at 15:51 -0700, Asif Lodhi wrote:
> 
(snip)

> > heavily used) part of the database completely in the memory (I am
> > talking about the in-memory database).
> 
> That's pretty typical, though I expect Oracle offers the admin more
> control over it than most do.
> 

It certainly does.  The Oracle 9i Server I talked about was a
high-load read-only MIS (extremely complex [IRS] reporting server, in
fact) and, in Oracle 9i, you can provide the datab ase server the
hints about a particular data table/partion (which is accessed the
most) so that Oracle 9i makes a best effort to keep the whole stuff
in-memory so that mechanical disk-access can be eliminated altogether
and SQL queries can access the data directly from server RAM (it's
totally transparent).  I mentioned it because I know MySQL has similar
in-memory database features (and one of my friends is using it to send
more than 800,000 banner ads a day for one of his company's clients -
albeit on a two-node itanium cluster and RAM, of course, in GB units)
amd MySQL supports pluggable data engines and one of the active ones
is Berkley DB.

My point was:  since MySQL supports in-memory databases and it also
uses Berkley DB as one of its engines, it is worth investigating
whether BDB+SVN can be re-configured to support in-memory data-stores
- to cut-down disk-contention and eliminate the slow-down experienced
and mentioned by some on this thread.

> > Though I haven't used SVN yet

(snip)

> 
> Regarding bdb: *hiss* *spit* . There is no other piece of software that
> has caused me more problems and data loss as a sysadmin. Recent versions
> can be very reliable if built with transaction support and if the apps
> use that transaction support, but some distros even now build BDB
> without transaction support. 

:)  Building BDB with or without transaction support is entirely your
prerogative.  Transaction _are_ costly in terms of their resource
overhead and should not be implemented if there is no need.

> BDB is also very vulnerable to issues
> caused by even small corruptions due to disk issues, hard poweroffs in
> the middle of a write, etc.

I took a look at BDB (BDB Java Edition, specifically - because of its
cross-platform attraction)  a couple of months back with reference to
the development of a high transaction volume stock-brokerage software.
 The version does seem to have some stability issues but I don't think
the more BDB (the 'C' version) has stability issues of any kind at
all.  I also conversed with their VP Business Development for quite
some time and he gave me references of some very solid BDB 
implementation sites (Nasdaq, Chicago Mercantile, etc) that are
_extremely_ high-load environments.  Besides, BDB (as their website
says) is being used by _big_ companies under heavy load.

> 
> BDB can work great, but it's really hard to get right. The 700 page book
> on the shelf behind me - the Berkley DB manual - reinforces that
> view ;-)

The BDB project that I am talking about never materialized (though I
pushed very hard and tried my level best to use an open-source data
management platform :) as my executive management didn't want to
plunge into something they weren't comfortable with.

> 
> I think Berkley DB is a fantastic tool for keeping a working store of
> data, so long as you keep the data in another format (presumably one
> harder to edit on the fly, etc) somewhere else.
> 
I think if you implement transaction support then you can eliviate
data loss problems.  BDB is just a libraray and, IMHO, it can be used
to develop systems customized to the exact specs -
application-specific data-management solutions, for example.

> SQLite seems to be showing up more and more as a potential alternative
> to BDB, and I find it more attractive myself. One advantage of using
> SQLite is that if you need "bigger" DB functionality later, or need to
> hook into a db like oracle or PostgreSQL with proper concurrent
> transaction support etc, you can do so much more easily.
> 
But SQLite is not being used by the kind of extremely high-profile
business entities whereas  BDB is being used by Sun, Motorola, Cisco,
Nasdaq and so many more.  Just take a look at their client list
(www.sleepycat.com).  IMHO, SQLite is currently good only for
average-load environments.

> I do think that however it's done, putting the Scribus file into a
> database and using the database's concurrency and locking features would
> be an interesting way to do multiple-user editing. A *huge* amount of
> work, I think, but potentially saner than most of the other approaches I
> can think of. Whether each copy of Scribus would talk to the db direct,
> or whether they'd use a "scribus document server" that knew all the db
> specific bits, I don't know.
> 

But that way, you'll be putting unstructured data (plain text) into a
structured database - well-suited for table/record oriented data.  Do
you envisage making the text-units (paragraphs, text-frame data, etc)
as records of the database -- (enabling per-recod, per paragraph, per
text-frame locking/concurrency while letting the other users work
simultaneously on other paragraphs of the same Scribus file)?

If you modify SVN and use BDB then you can use it the way you like. 
BDB is not a strictly structured RDBMS - it is just a data management
library - that's why SVN uses it to manage simultaneous access to
un-structured textual data (the source code, that is).  You can use it
as a data-management engine of your RDBMS to store highly structured
data (like MySQL does) or as a data-management engine of a
version-control system to manage simultaneous access to unstructured
data such as text (like Subversion does).

The reason I am stressing BDB (or any open and effective data
management library) is that I think the only way you can use a
database with Scribus (or any DTP for that matter) is to break textual
data into record-like data units and store/manage text in the database
on a per-record basis.  Do you really think that various elements
typical of an average DTP document editing (layout, text-writing,
graphics, etc.) would allow you to discover clean abstractions that
you can cleanly fit into database records?  It's just my humble
opinion!  I am not one of  the Scribus developers - you guys are.  I
just can't see how the unstructured information can be made to fit in
a structured RDBMS.  I think it can be generic data-store like BDB -
not a structured RDBMS.

> The whole document-in-database thing isn't a new discussion. Like most
> of the things discussed here, what we really need is someone to say "I
> have the time and skills, or the time and am willing to learn the
> skills, to do this. Where do I start?" These folks, alas, do not grow on
> trees.
> 

IIRC, someone on this thread also talked about integrating the DTP
workflow with the data-store/database - that really sounds
interesting.  I would like to you elaborate more on it.

(snip)

> flat file store as an alternative because so many people just wouldn't
> trust bdb (myself included), too.

Then, why BDB is being used by so many high-profile companies like
Google, Motorola, etc.?

(snip)

Thanks for the excellent response.

--
Best regards

Asif