From khaledhosny at eglug.org  Mon May  2 16:46:32 2016
From: khaledhosny at eglug.org (Khaled Hosny)
Date: Mon, 2 May 2016 18:46:32 +0200
Subject: [scribus-dev] Support for high DPI screens
Message-ID: <20160502164632.GA25223@macbook>

Hi,

I?m a bit interested in improving support for high DPI screens (such my
main machine has one). I recently sent a couple of patches to help
improve things:

* https://bugs.scribus.net/view.php?id=14004
  This improves the canvas rendering so that it does not very blurry on
  hi-DPI screens. AFAICT this should compile on any Qt 5 version (at
  least the used APIs has no version-specific annotations).

* https://bugs.scribus.net/view.php?id=13204
  This enables hi-DPI support without needing to set an env variable on
  X11 (my understanding that Qt on Windows, Mac OS X and Wayland enables
  hi-DPI automatically if the system supports it). This works only with
  Qt 5.6, and the code is #if?d as such, for older systems one has to
  manually set QT_DEVICE_PIXEL_RATIO env variable.

The most prominent hi-DPI issue remaining is the icons, currently Qt wil
just scslae them so they appear blurry. There are two ways to fix this,
use SVG icons or have two set of the icons (normal and hi-DPI) and the
code automatically selects between them. Either way if someone is
welling to work on the icon part, I can look into the code part of
loading the icon appropriately.

Regards,
Khaled


From rob.dixon at erros.co.uk  Tue May 10 08:06:47 2016
From: rob.dixon at erros.co.uk (rob.dixon at erros.co.uk)
Date: Tue, 10 May 2016 09:06:47 +0100 (BST)
Subject: [scribus-dev] FormatXML
Message-ID: <351774625.261849.2de72a91-d91c-42c7-8598-2e21a87ed1b6.open-xchange@webmail.123-reg.co.uk>

An HTML attachment was scrubbed...
URL: <http://lists.scribus.net/pipermail/scribus-dev/attachments/20160510/7d334746/attachment.html>

From wtxnh-scribus at yahoo.com.au  Tue May 10 10:18:22 2016
From: wtxnh-scribus at yahoo.com.au (Juraj Fedel)
Date: Tue, 10 May 2016 12:18:22 +0200
Subject: [scribus-dev] FormatXML
In-Reply-To: <351774625.261849.2de72a91-d91c-42c7-8598-2e21a87ed1b6.open-xchange@webmail.123-reg.co.uk>
References: <351774625.261849.2de72a91-d91c-42c7-8598-2e21a87ed1b6.open-xchange@webmail.123-reg.co.uk>
Message-ID: <20160510101822.GA1003@gazda>

On Tue, May 10, 2016 at 09:06:47AM +0100, rob.dixon at erros.co.uk wrote:
> I wish to import XML into Scribus.
> I have tried to create a Python script FormatXML, taken from
> the page https://wiki.scribus.net/canvas/Formatxml

The script on wikipage is somewhat outdated, probably written for older
version of scibus scripter. I have made minimal changes I could for
script to run on current 1.5.2-svn version of scribus.

After runing the script with sample xml from wikipage I noticed that
sla:override attribute does not work as I would expect. In sample xml it
should apply font override only to word 'consectetur'. Instead it is
applied to the rest of current paragraph too. Me not being good at
parsing xml document will leave to somebody else to fix this behavior.

Bye
Juraj Fedel
-------------- next part --------------
A non-text attachment was scrubbed...
Name: importxml.py
Type: text/x-python
Size: 6444 bytes
Desc: not available
URL: <http://lists.scribus.net/pipermail/scribus-dev/attachments/20160510/eedfa945/attachment.py>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: import_me.xml
Type: application/xml
Size: 389 bytes
Desc: not available
URL: <http://lists.scribus.net/pipermail/scribus-dev/attachments/20160510/eedfa945/attachment.wsdl>

From jluc at no-log.org  Tue May 10 10:42:39 2016
From: jluc at no-log.org (JLuc)
Date: Tue, 10 May 2016 12:42:39 +0200
Subject: [scribus-dev] FormatXML
In-Reply-To: <20160510101822.GA1003@gazda>
References: <351774625.261849.2de72a91-d91c-42c7-8598-2e21a87ed1b6.open-xchange@webmail.123-reg.co.uk>
 <20160510101822.GA1003@gazda>
Message-ID: <ngsduv$90k$1@ger.gmane.org>

Hello Juraj,

Have you updated the script on the wiki ?
It would be good since its useless to leave a bad version when it doesnt run anymore.

It looks like script need to be "versionned" : could you also add a comment "tested ok for 1.5.2svn" ?

JL


Le 10/05/2016 12:18, Juraj Fedel a ?crit :
>
> #!/usr/bin/env python
> # -*- coding: utf-8 -*-
>
> import sys
>
> try:
>     # Please do not use 'from scribus import *' . If you must use a 'from import',
>     # Do so_after_  the 'import scribus' and only import the names you need, such
>     # as commonly used constants.
>     import scribus
> except ImportError, err:
>     print "This Python script is written for the Scribus scripting interface."
>     print "It can only be run from within Scribus."
>     sys.exit(1)
>
> import xml.sax
> import xml.sax.handler
>
> class XMLFormatter (xml.sax.handler.ContentHandler):
>     """Formats xml in scribus document
>     Every text node including whitespace is placed in the document
>     Paragraph styles are applied through an attribute sla:style in the namespace"http://www.scribus.net/formatxml".
>     Character styles are applied through the attribute sla:override in the same namespace
>     Paragraph and character styles are inherited automatically"""
>
>     def __init__ (self):
>         self.styles = [None]
>         self.overrides = [None]
>         self.template = ""
>         self.document = ""
>         self.name = ""
>         self.in_namespace = False
>         self.prefix = None
>         self.first = True
>
>     def make_textframe (self):
>         margins = scribus.getPageMargins()
>         size = scribus.getPageSize()
>         w = size[0] - margins[1] - margins[3]
>         h = size[1] - margins[0] - margins[2]
>         return scribus.createText (margins[1], margins[0], w, h)
>
>     def startPrefixMapping(self, prefix, uri):
>         """checks if current node is in proper namespace"""
>         if uri =="http://www.scribus.net/formatxml":
>             self.in_namespace = True
>             if prefix != "":
>                 self.prefix = prefix + ":"
>
>     def endPrefixMapping(self, prefix):
>         """signal that we are no longer in namespace"""
>         if self.in_namespace:
>             self.in_namespace = False
>             self.prefix = None
>
>     def startDocument(self):
>         """Open document"""
>         scribus.openDoc(self.template)
>         self.name = self.make_textframe()
>
>     def add_style (self, stack, style):
>         if style == None:
>             style = stack[-1]
>         stack.append(style)
>
>     def startElementNS(self, name, qname, attrs):
>         """record styles or overrides"""
>         attrs_names = attrs.getQNames()
>         if self.in_namespace:
>             if self.prefix + "style" in attrs_names:
>                 style = attrs.getValueByQName(self.prefix + "style")
>             else:
>                 style = None
>             self.add_style(self.styles, style)
>             if self.prefix + "override" in attrs_names:
>                 override = attrs.getValueByQName(self.prefix + "override")
>             else:
>                 override = None
>             self.add_style(self.overrides, override)
>
>     def endElementNS(self, name, qname):
>         self.styles.pop()
>         self.overrides.pop()
>
>     def shouldSetStyle (self):
>         if self.first:
>             return True
>         else:
>             return scribus.getAllText(self.name)[-1] == "\r"
>
>     def characters(self, content):
>         """place text, apply style"""
>         start = scribus.getTextLength(self.name)
>         go_ahead = self.shouldSetStyle()
>         scribus.insertText (content, -1, self.name)
>         scribus.selectText(start, len(content), self.name)
>         if self.styles[-1] != None and go_ahead:
>             try:
>                 scribus.setStyle(self.styles[-1], self.name)
>             except scribus.NotFoundError:
>                 scribus.createParagraphStyle(self.styles[-1])
>                 scribus.setStyle(self.styles[-1], self.name)
>         if self.overrides[-1] != None:
>             try:
>                 scribus.setFont(self.overrides[-1], self.name)
>             except ValueError:
>                 pass
>         self.first = False
>
>     def flow(self):
>         while(scribus.textOverflows(self.name) > 0 and scribus.getTextLines(self.name)):
>             current = self.name
>             scribus.newPage(-1)
>             scribus.gotoPage( scribus.pageCount() )
>             self.name = self.make_textframe()
>             scribus.linkTextFrames(current, self.name)
>
>     def endDocument(self):
>         """Save and close document"""
>         self.flow()
>         scribus.saveDocAs(self.document)
>         scribus.closeDoc()
> #end class XMLFormatter
>
> class UserCanceled (Exception):
>     pass
> #end class UserCanceled
>
> def format(xml_file, scribus_template, document):
>     fmt = XMLFormatter ()
>     fmt.template = scribus_template
>     fmt.document = document
>     p = xml.sax.make_parser()
>     p.setFeature(xml.sax.handler.feature_namespaces, True)
>     p.setContentHandler(fmt)
>     p.parse(xml_file)
>
> def getFile(caption, filter, defaultname=None):
>     if defaultname:
>        file = scribus.fileDialog(caption, filter=filter, defaultname=defaultname, issave=True)
>     else:
>        file = scribus.fileDialog(caption, filter=filter)
>     if len(file) == 0:
>         raise UserCanceled("canceled")
>     else:
>         return file
>
> def main():
>     """Formats xml file using Scribus template"""
>     try:
>         xml_file = getFile("XML File", 'XML (*.xml)')
>         template = getFile("Scribus template", "Scribus document (*.sla)")
>         document = getFile("Save document", "All (*.*)", xml_file.replace(".xml", ".sla"))
>         format (xml_file, template, document)
>     except UserCanceled:
>         pass
>     except xml.sax.SAXParseException:
>         scribus.messageBox("XML Error", "XML Error: please check your document")
>
> def main_wrapper():
>     """The main_wrapper() function disables redrawing, sets a sensible generic
>     status bar message, and optionally sets up the progress bar. It then runs
>     the main() function. Once everything finishes it cleans up after the main()
>     function, making sure everything is sane before the script terminates."""
>     try:
>         scribus.statusMessage("Running script...")
>         scribus.progressReset()
>         main()
>     finally:
>         # Exit neatly even if the script terminated with an exception,
>         # so we leave the progress bar and status bar blank and make sure
>         # drawing is enabled.
>         if scribus.haveDoc():
>             scribus.setRedraw(True)
>         scribus.statusMessage("")
>         scribus.progressReset()
>
> # This code detects if the script is being run as a script, or imported as a module.
> # It only runs main() if being run as a script. This permits you to import your script
> # and control it manually for debugging.
> if __name__ == '__main__':
>     main_wrapper()




From wtxnh-scribus at yahoo.com.au  Tue May 10 11:37:30 2016
From: wtxnh-scribus at yahoo.com.au (Juraj Fedel)
Date: Tue, 10 May 2016 13:37:30 +0200
Subject: [scribus-dev] FormatXML
In-Reply-To: <ngsduv$90k$1@ger.gmane.org>
References: <351774625.261849.2de72a91-d91c-42c7-8598-2e21a87ed1b6.open-xchange@webmail.123-reg.co.uk>
 <20160510101822.GA1003@gazda> <ngsduv$90k$1@ger.gmane.org>
Message-ID: <20160510113730.GB1003@gazda>

On Tue, May 10, 2016 at 12:42:39PM +0200, JLuc wrote:
> Have you updated the script on the wiki ?

No, sorry. I do not have wiki account.
And this would be mine first editing of wiki ever, so I prefer
somebody else do it instead :-)

Juraj Fedel


From rob.dixon at erros.co.uk  Tue May 10 12:18:49 2016
From: rob.dixon at erros.co.uk (Rob Dixon)
Date: Tue, 10 May 2016 13:18:49 +0100
Subject: [scribus-dev] FormatXML
In-Reply-To: <20160510101822.GA1003@gazda>
References: <351774625.261849.2de72a91-d91c-42c7-8598-2e21a87ed1b6.open-xchange@webmail.123-reg.co.uk>
 <20160510101822.GA1003@gazda>
Message-ID: <98DDA968-A5FF-4F70-8BC7-FE1061501D4D@erros.co.uk>

Hi Juraj,

Many thanks, but unfortunately I haven?t managed to download 1.5.2 as it is not the current stable version which is 1.4.6. I did manage to download 1.5,1 and tried to use the source code with that but I got the same error. It maybe that you haven?t updated the script on the wiki or your changes don?t work with 1.5.1. I hope that this can be resolved as I really would like to use Scribus using imported XML

Best wishes

Rob 

> On 10 May 2016, at 11:18, Juraj Fedel <wtxnh-scribus at yahoo.com.au> wrote:
> 
> On Tue, May 10, 2016 at 09:06:47AM +0100, rob.dixon at erros.co.uk wrote:
>> I wish to import XML into Scribus.
>> I have tried to create a Python script FormatXML, taken from
>> the page https://wiki.scribus.net/canvas/Formatxml
> 
> The script on wikipage is somewhat outdated, probably written for older
> version of scibus scripter. I have made minimal changes I could for
> script to run on current 1.5.2-svn version of scribus.
> 
> After runing the script with sample xml from wikipage I noticed that
> sla:override attribute does not work as I would expect. In sample xml it
> should apply font override only to word 'consectetur'. Instead it is
> applied to the rest of current paragraph too. Me not being good at
> parsing xml document will leave to somebody else to fix this behavior.
> 
> Bye
> Juraj Fedel
> <importxml.py><import_me.xml>_______________________________________________
> scribus-dev mailing list
> scribus-dev at lists.scribus.net
> http://lists.scribus.net/mailman/listinfo/scribus-dev



From wtxnh-scribus at yahoo.com.au  Tue May 10 14:06:15 2016
From: wtxnh-scribus at yahoo.com.au (Juraj Fedel)
Date: Tue, 10 May 2016 16:06:15 +0200
Subject: [scribus-dev] FormatXML
In-Reply-To: <98DDA968-A5FF-4F70-8BC7-FE1061501D4D@erros.co.uk>
References: <351774625.261849.2de72a91-d91c-42c7-8598-2e21a87ed1b6.open-xchange@webmail.123-reg.co.uk>
 <20160510101822.GA1003@gazda>
 <98DDA968-A5FF-4F70-8BC7-FE1061501D4D@erros.co.uk>
Message-ID: <20160510140615.GC1003@gazda>

On Tue, May 10, 2016 at 01:18:49PM +0100, Rob Dixon wrote:

On Tue, May 10, 2016 at 01:18:49PM +0100, Rob Dixon wrote:
> ... I haven?t managed to download 1.5.2 ...

I included script as attachment in my original mail (and in reply by
JLuc it is included in the body of mail).

I now tested it with scribus 1.4.6 and script work with that version too.
I hope you will be able to use the script successfully.

> > On 10 May 2016, at 11:18, Juraj Fedel <wtxnh-scribus at yahoo.com.au> wrote:
> > After runing the script with sample xml from wikipage I noticed that
> > sla:override attribute does not work as I would expect. In sample xml it
> > should apply font override only to word 'consectetur'. Instead it is
> > applied to the rest of current paragraph too. Me not being good at
> > parsing xml document will leave to somebody else to fix this behavior.

When I run script with 1.4.6 version I see that sla:override attribute
is handled correctly ie. it is applied only to one word 'consectetur'.
That makes me wonder why 1.5.2svn misbehave in this case.

Juraj Fedel


From jluc at no-log.org  Tue May 10 15:44:49 2016
From: jluc at no-log.org (JLuc)
Date: Tue, 10 May 2016 17:44:49 +0200
Subject: [scribus-dev] FormatXML
In-Reply-To: <20160510113730.GB1003@gazda>
References: <351774625.261849.2de72a91-d91c-42c7-8598-2e21a87ed1b6.open-xchange@webmail.123-reg.co.uk>
 <20160510101822.GA1003@gazda> <ngsduv$90k$1@ger.gmane.org>
 <20160510113730.GB1003@gazda>
Message-ID: <ngsvli$sgf$1@ger.gmane.org>

Le 10/05/2016 13:37, Juraj Fedel a ?crit :
> On Tue, May 10, 2016 at 12:42:39PM +0200, JLuc wrote:
>> Have you updated the script on the wiki ?
>
> No, sorry. I do not have wiki account.
> And this would be mine first editing of wiki ever, so I prefer
> somebody else do it instead :-)

Nice to read its ok on 1.4.6 also.

As for the wiki, you'd find out it's much easyer
than debuging python scripts !
Register here :
https://wiki.scribus.net/wiki/index.php?title=Special:UserLogin&returnto=Scribus
then i believe you'll have to wait a bit till some authorized person grants you access
and then, well, follow the links !
Go to the page, click "Edit", select old script, paste yours instead, etc...

JLuc




From rob.dixon at erros.co.uk  Tue May 10 15:47:23 2016
From: rob.dixon at erros.co.uk (rob.dixon at erros.co.uk)
Date: Tue, 10 May 2016 16:47:23 +0100 (BST)
Subject: [scribus-dev] FormatXML
In-Reply-To: <20160510140615.GC1003@gazda>
References: <351774625.261849.2de72a91-d91c-42c7-8598-2e21a87ed1b6.open-xchange@webmail.123-reg.co.uk>
 <20160510101822.GA1003@gazda>
 <98DDA968-A5FF-4F70-8BC7-FE1061501D4D@erros.co.uk>
 <20160510140615.GC1003@gazda>
Message-ID: <573579637.16129.6545fec5-d688-4424-8571-56d28d561f0a.open-xchange@webmail.123-reg.co.uk>

Juraj,

Many thanks. I didn't spot your attachment before. I have now used this and I
can now import XML into Scribus. As I know nothing about Scribus, it will take
me a little while to learn how to use it, but at least I can get the data in.

Best wishes

Rob

> 
>     On 10 May 2016 at 15:06 Juraj Fedel <wtxnh-scribus at yahoo.com.au> wrote:
> 
> 
>     On Tue, May 10, 2016 at 01:18:49PM +0100, Rob Dixon wrote:
> 
>     On Tue, May 10, 2016 at 01:18:49PM +0100, Rob Dixon wrote:
> 
>         > > 
> >         ... I haven?t managed to download 1.5.2 ...
> > 
> >     > 
>     I included script as attachment in my original mail (and in reply by
>     JLuc it is included in the body of mail).
> 
>     I now tested it with scribus 1.4.6 and script work with that version too.
>     I hope you will be able to use the script successfully.
> 
>         > > 
> >         > On 10 May 2016, at 11:18, Juraj Fedel <wtxnh-scribus at yahoo.com.au>
> >         > wrote:
> > 
> >             > > > 
> > >             After runing the script with sample xml from wikipage I
> > > noticed that
> > >             sla:override attribute does not work as I would expect. In
> > > sample xml it
> > >             should apply font override only to word 'consectetur'. Instead
> > > it is
> > >             applied to the rest of current paragraph too. Me not being
> > > good at
> > >             parsing xml document will leave to somebody else to fix this
> > > behavior.
> > > 
> > >         > > 
> >         When I run script with 1.4.6 version I see that sla:override
> > attribute
> > 
> >     > 
>     is handled correctly ie. it is applied only to one word 'consectetur'.That
> makes me wonder why 1.5.2svn misbehave in this case.
> 
>     Juraj Fedel
> 
>     _______________________________________________
>     scribus-dev mailing list
>     scribus-dev at lists.scribus.net
>     http://lists.scribus.net/mailman/listinfo/scribus-dev
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.scribus.net/pipermail/scribus-dev/attachments/20160510/8e7e10ea/attachment.html>

From scribus.user at gmail.com  Thu May 12 23:09:39 2016
From: scribus.user at gmail.com (Scribus User)
Date: Thu, 12 May 2016 18:09:39 -0500
Subject: [scribus-dev] Requesting help to update Travis-CI to use Docker
Message-ID: <CAAt0FRjMEYcjEm850hQqLn2DC4k0rjaB4+OtB4-WqC+m1O0Q+A@mail.gmail.com>

Hey folks,

Can anyone spearhead fixing our current CI setup. Since Scribus trunk was
bumped to support Qt5.3 we've had issues with building Scribus on Ubuntu
Trusty via Travis-CI. The solution is to use Travis-CI + Docker to create a
custom build environment: https://docs.travis-ci.com/user/docker/

Scribus's:
- Travis-CI page: https://travis-ci.org/scribusproject/scribus
- SVN github mirror (that triggers TravisCI):
https://github.com/scribusproject/scribus
- .travis.yml file:
https://github.com/scribusproject/scribus/blob/master/.travis.yml

If you feel inspired to help with this feel free to contact me directly via
my nick at scribus dot net

Cheers,
kunda
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.scribus.net/pipermail/scribus-dev/attachments/20160512/2f28046c/attachment.html>

From khaledhosny at eglug.org  Thu May 12 23:17:09 2016
From: khaledhosny at eglug.org (Khaled Hosny)
Date: Fri, 13 May 2016 01:17:09 +0200
Subject: [scribus-dev] Requesting help to update Travis-CI to use Docker
In-Reply-To: <CAAt0FRjMEYcjEm850hQqLn2DC4k0rjaB4+OtB4-WqC+m1O0Q+A@mail.gmail.com>
References: <CAAt0FRjMEYcjEm850hQqLn2DC4k0rjaB4+OtB4-WqC+m1O0Q+A@mail.gmail.com>
Message-ID: <20160512231709.GB8040@macbook>

On Thu, May 12, 2016 at 06:09:39PM -0500, Scribus User wrote:
> Hey folks,
> 
> Can anyone spearhead fixing our current CI setup. Since Scribus trunk was
> bumped to support Qt5.3 we've had issues with building Scribus on Ubuntu
> Trusty via Travis-CI. The solution is to use Travis-CI + Docker to create a
> custom build environment: https://docs.travis-ci.com/user/docker/

This is how I foxed it:
https://github.com/HOST-Oman/scribus/commit/bbaaa349f4c059bf5fa37140c3922362e7f39e79

Not the best way (using an unofficial PPA), but that is the least
invasive fix.

Regards,
Khaled


From scribus.user at gmail.com  Fri May 13 01:31:20 2016
From: scribus.user at gmail.com (Scribus User)
Date: Thu, 12 May 2016 20:31:20 -0500
Subject: [scribus-dev] Requesting help to update Travis-CI to use Docker
In-Reply-To: <20160512231709.GB8040@macbook>
References: <CAAt0FRjMEYcjEm850hQqLn2DC4k0rjaB4+OtB4-WqC+m1O0Q+A@mail.gmail.com>
 <20160512231709.GB8040@macbook>
Message-ID: <CAAt0FRitQPdE4fpyK2kNGJWLquohH4D_GygAZERYeS_h+wKHSA@mail.gmail.com>

On Thu, May 12, 2016 at 6:17 PM, Khaled Hosny <khaledhosny at eglug.org> wrote:

> On Thu, May 12, 2016 at 06:09:39PM -0500, Scribus User wrote:
> > Hey folks,
> >
> > Can anyone spearhead fixing our current CI setup. Since Scribus trunk was
> > bumped to support Qt5.3 we've had issues with building Scribus on Ubuntu
> > Trusty via Travis-CI. The solution is to use Travis-CI + Docker to
> create a
> > custom build environment: https://docs.travis-ci.com/user/docker/
>
> This is how I foxed it:
>
> https://github.com/HOST-Oman/scribus/commit/bbaaa349f4c059bf5fa37140c3922362e7f39e79
>
> Not the best way (using an unofficial PPA), but that is the least
> invasive fix.
>
> Regards,
> Khaled
>

Khaled,
Great temporary fix!
Devs, here is a patch:
https://github.com/HOST-Oman/scribus/commit/bbaaa349f4c059bf5fa37140c3922362e7f39e79.patch


Thanks!
/Kunda
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.scribus.net/pipermail/scribus-dev/attachments/20160512/26b2dc22/attachment.html>

From scribus.user at gmail.com  Fri May 13 16:33:46 2016
From: scribus.user at gmail.com (Scribus User)
Date: Fri, 13 May 2016 11:33:46 -0500
Subject: [scribus-dev] Is there a way to have multiple versions of Scribus
 on Windows 10?
Message-ID: <CAAt0FRgaw3-HmonyeBJbrjWi2u56DC4Hv_c0H8xcxjmgQbmKow@mail.gmail.com>

Issue mentioned by Garry Prachett in
https://bugs.scribus.net/view.php?id=13989#c40575
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.scribus.net/pipermail/scribus-dev/attachments/20160513/9fc629f8/attachment.html>

From christoph-schaefer at gmx.de  Tue May 17 20:51:25 2016
From: christoph-schaefer at gmx.de (=?UTF-8?Q?=22Christoph_Sch=C3=A4fer=22?=)
Date: Tue, 17 May 2016 22:51:25 +0200
Subject: [scribus-dev] Scribus 1.5.2 released
Message-ID: <trinity-8a6b7a59-86f1-47c8-b492-df8e6afe00a1-1463518285535@3capp-gmx-bs14>

The Scribus Team is very pleased to announce the release of the development version Scribus 1.5.2, which is another major step onto the next fully-featured version 1.6.0. In terms of stability, 1.5.2 can be already be regarded as robust. The Scribus Team encourages the wide-spread use and testing of version 1.5.2 in as many environments as possible. User feedback will help us to release a rock-solid version 1.6.0.


Most Important Changes
======================

- The text layout engine has been rewritten from scratch in preparation for support of complex scripts such as Arabic, Hebrew, Chinese and Hindi coming in Scribus 1.5.3 and later. We are more than grateful for the magnificent work that the Oman House of Open Source Technology team, led by Khaled Hosny, completed.

- Within the context of the text layout system rewrite, some new text features introduced with the 1.5.0 release have been stabilized and improved.

- Improvements to the canvas rendering on Hi-DPI screens.

- The Autosave and File Recovery system has seen large improvements and is now highly configurable.

- The Resource Manager, as well as the official Scribus Resources pool have been expanded significantly, because Scribus 1.5.2 provides direct access to the more than 300 (mostly commercial) color palettes of the Open Colour Systems Collection in the LAB color space. We would like to thank dtp studio oldenburg and the initiative freieFarbe for making these color palettes available under a CC license. The Resource Manager user interface has received some enhancements.
    
- The Resource Manager now verifies resource downloads from Scribus servers with SHA256 checksums.


A complete changelog is available here: https://bugs.scribus.net/view_all_bug_page.php?filter=161020

A full data sheet (PDF) is available here: https://wiki.scribus.net/wiki/images/9/93/Scribus-specs-152.pdf


Caveats
=======

- Please note that the online help system hasn't been updated yet, and it probably be won't be until the final release of Scribus 1.6.0. However, with Scribus 1.5.2, Scribus now supports downloading of the online help manual and as this work is completed within Scribus 1.5.3, we will decouple the release of the help manual from the main application which will allow more regular updates to the help system.
    
- Some newly introduced features since 1.5.0, like footnotes, tables, ordered/un-ordered lists are "program-stable", i.e., they won't crash Scribus 1.5.1, but not "feature-stable", which means they may sometimes not work as expected for the time being.

- There will be more wide sweeping changes coming in 1.5.3 relating to text system updates for the complex language system support. People using Scribus 1.5.1 or 1.5.2 should ensure they keep document backups and the old versions installed prior to testing 1.5.3.svn and the 1.5.3 release when it arrives.


Dependencies
============

- Qt 5.3 or higher is now required

- cairo 1.14 or higher is now required
    
- Other dependencies will be reviewed in 1.5.3



Primary Download Locations
==========================

    
- Installation packages for Windows, Mac OS X, a Linux AppImage and the source code are available here: https://sourceforge.net/projects/scribus/files/scribus-devel/1.5.2/
    
- Fedora and CentOS RPMs: http://download.opensuse.org/repositories/home:/mrdocs
    
- OpenSUSE, SLED, and SLES RPMs: http://download.opensuse.org/repositories/home:/mrdocs

- Packaging for other Linux distributions, *BSD, Solaris and OpenIndiana is beyond our influence. We recommend updating the respective repository data on a regular basis.


Download Verification
=====================

Please visit the official announcement on https://wiki.scribus.net/canvas/1.5.2_Release for Sha256 and Sha1 checksums.


Credits
=======


The Scribus Team would like to thank Anduin.net and Modirum for their continued hosting of all of the Scribus websites.

We are grateful to the Organisation Internationale de la Francophonie and Linux New Media (Germany) for sponsoring.

The Scribus Team is also honored to have Resene Colours (New Zealand), dtp studio Oldenburg (Germany), Scientific Illustration Services Corp. (USA), the Newspaper Association of America (USA), Software Consulting Services (USA) and freieFarbe (Germany) as Special Supporters and donors of color palettes and other content since the 1.4.x release, just like we are grateful to the owner of Vector Portal for the permission to distribute some of his work as Scribus Templates.

Porting Scribus to OS/2 and eComStation is being supported by Mensys BV (The Netherlands) and Serenity Systems (USA).

Finally, the Scribus Team would like to thank the many end users, translators, testers and contributors who helped us with this release.



An HTML version of this announcement with screenshots is availble here: https://wiki.scribus.net/canvas/1.5.2_Release


From andreas.vox at gmail.com  Mon May 30 15:26:54 2016
From: andreas.vox at gmail.com (=?utf-8?B?QW5kcmXNrmHNpnPNrw==?=)
Date: Mon, 30 May 2016 17:26:54 +0200
Subject: [scribus-dev] Change to UTF-32
Message-ID: <016001d1ba87$b2ee0670$18ca1350$@googlemail.com>

Hi!

Khaled recently committed a change to the HOST-Oman/ctl branch that changes the text coding from UTF_16 (QChar/QString) to UTF-32:
https://github.com/HOST-Oman/scribus/commit/b2886e64b4aba660e4939d6cdaba066aed39d081

Since we have divided opinions on this issue, would you mind reviewing the discussion in the comments and add your own thoughts?

Best regards
/Andreas



From cbradney at scribus.info  Mon May 30 18:56:06 2016
From: cbradney at scribus.info (Craig Bradney)
Date: Mon, 30 May 2016 20:56:06 +0200
Subject: [scribus-dev] Change to UTF-32
In-Reply-To: <016001d1ba87$b2ee0670$18ca1350$@googlemail.com>
References: <016001d1ba87$b2ee0670$18ca1350$@googlemail.com>
Message-ID: <3E21CCFD-A756-4D47-982D-4C691029AA21@scribus.info>


> On 30 May 2016, at 17:26, Andre?a?s? <andreas.vox at gmail.com> wrote:
> 
> Hi!
> 
> Khaled recently committed a change to the HOST-Oman/ctl branch that changes the text coding from UTF_16 (QChar/QString) to UTF-32:
> https://github.com/HOST-Oman/scribus/commit/b2886e64b4aba660e4939d6cdaba066aed39d081
> 
> Since we have divided opinions on this issue, would you mind reviewing the discussion in the comments and add your own thoughts?
> 
> Best regards
> /Andreas
> 
> 


I don?t really see the difference because I don?t know the code. If the APIs enable me, or anyone who doesn?t know the text code to do stuff like insert text at a certain point for example, the it would seem ok.
One thing that is mentioned is for example inserting a QChar.. so how does one programmatically insert 1 character?

I don?t have an opinion on the 16 vs 32 as long as it works so we can do all operations.. read/write to file, display it, type it, export it to various formats.

Is there a speed difference?

At the beginning of the conversation, you seem to disagree with Khaled and then at the end (where it is now) you disagree less?

Craig

From louis.desjardins at gmail.com  Mon May 30 19:01:58 2016
From: louis.desjardins at gmail.com (Louis Desjardins)
Date: Mon, 30 May 2016 15:01:58 -0400
Subject: [scribus-dev] Change to UTF-32
In-Reply-To: <3E21CCFD-A756-4D47-982D-4C691029AA21@scribus.info>
References: <016001d1ba87$b2ee0670$18ca1350$@googlemail.com>
 <3E21CCFD-A756-4D47-982D-4C691029AA21@scribus.info>
Message-ID: <CAM9PYkaqzObPk7NVmcr=kAMMv7X3Di4oMUS2kaJPmBPt4BPJ_w@mail.gmail.com>

2016-05-30 14:56 GMT-04:00 Craig Bradney <cbradney at scribus.info>:

>
> > On 30 May 2016, at 17:26, Andre?a?s? <andreas.vox at gmail.com> wrote:
> >
> > Hi!
> >
> > Khaled recently committed a change to the HOST-Oman/ctl branch that
> changes the text coding from UTF_16 (QChar/QString) to UTF-32:
> >
> https://github.com/HOST-Oman/scribus/commit/b2886e64b4aba660e4939d6cdaba066aed39d081
> >
> > Since we have divided opinions on this issue, would you mind reviewing
> the discussion in the comments and add your own thoughts?
> >
> > Best regards
> > /Andreas
> >
> >
>
>
> I don?t really see the difference because I don?t know the code. If the
> APIs enable me, or anyone who doesn?t know the text code to do stuff like
> insert text at a certain point for example, the it would seem ok.
> One thing that is mentioned is for example inserting a QChar.. so how does
> one programmatically insert 1 character?
>
> I don?t have an opinion on the 16 vs 32 as long as it works so we can do
> all operations.. read/write to file, display it, type it, export it to
> various formats.
>

Does this matter at text importation?

Louis

>
> Is there a speed difference?
>
> At the beginning of the conversation, you seem to disagree with Khaled and
> then at the end (where it is now) you disagree less?
>
> Craig
> _______________________________________________
> scribus-dev mailing list
> scribus-dev at lists.scribus.net
> http://lists.scribus.net/mailman/listinfo/scribus-dev
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.scribus.net/pipermail/scribus-dev/attachments/20160530/f5f5daa0/attachment.html>

From khaledhosny at eglug.org  Mon May 30 19:14:28 2016
From: khaledhosny at eglug.org (Khaled Hosny)
Date: Mon, 30 May 2016 21:14:28 +0200
Subject: [scribus-dev] Change to UTF-32
In-Reply-To: <3E21CCFD-A756-4D47-982D-4C691029AA21@scribus.info>
References: <016001d1ba87$b2ee0670$18ca1350$@googlemail.com>
 <3E21CCFD-A756-4D47-982D-4C691029AA21@scribus.info>
Message-ID: <20160530191428.GA20467@macbook>

On Mon, May 30, 2016 at 08:56:06PM +0200, Craig Bradney wrote:
> 
> > On 30 May 2016, at 17:26, Andre?a?s? <andreas.vox at gmail.com> wrote:
> > 
> > Hi!
> > 
> > Khaled recently committed a change to the HOST-Oman/ctl branch that changes the text coding from UTF_16 (QChar/QString) to UTF-32:
> > https://github.com/HOST-Oman/scribus/commit/b2886e64b4aba660e4939d6cdaba066aed39d081
> > 
> > Since we have divided opinions on this issue, would you mind reviewing the discussion in the comments and add your own thoughts?
> > 
> > Best regards
> > /Andreas
> > 
> > 
> 
> 
> I don?t really see the difference because I don?t know the code. If
> the APIs enable me, or anyone who doesn?t know the text code to do
> stuff like insert text at a certain point for example, the it would
> seem ok.
> One thing that is mentioned is for example inserting a QChar.. so how
> does one programmatically insert 1 character?

StoryText has no API to insert single character, inserting QChar
apparently works because it can be cast to QString or some such. You can
now either insert a QString and the code will convert it to UTF-32, or a
QVector<uint> of UTF-32 code points. We might have a friendly typedef
for this vector.

> I don?t have an opinion on the 16 vs 32 as long as it works so we can
> do all operations.. read/write to file, display it, type it, export it
> to various formats.

The big difference is that UTF-16 is variable length, a Unicode
character can be represented by one or two UTF-16 code units, but
Scribus was treating it as a fixed length encoding which is wrong.
UTF-32, on the other hand, is fixed length encoding, so the assumption
all over Scribus codebase can be kept.

Regards,
Khaled


From cbradney at scribus.info  Mon May 30 19:22:43 2016
From: cbradney at scribus.info (Craig Bradney)
Date: Mon, 30 May 2016 21:22:43 +0200
Subject: [scribus-dev] Change to UTF-32
In-Reply-To: <20160530191428.GA20467@macbook>
References: <016001d1ba87$b2ee0670$18ca1350$@googlemail.com>
 <3E21CCFD-A756-4D47-982D-4C691029AA21@scribus.info>
 <20160530191428.GA20467@macbook>
Message-ID: <FC43BF8B-9CD6-438E-94B0-DDC2631D0941@scribus.info>


> On 30 May 2016, at 21:14, Khaled Hosny <khaledhosny at eglug.org> wrote:
> 
> On Mon, May 30, 2016 at 08:56:06PM +0200, Craig Bradney wrote:
>> 
>>> On 30 May 2016, at 17:26, Andre?a?s? <andreas.vox at gmail.com> wrote:
>>> 
>>> Hi!
>>> 
>>> Khaled recently committed a change to the HOST-Oman/ctl branch that changes the text coding from UTF_16 (QChar/QString) to UTF-32:
>>> https://github.com/HOST-Oman/scribus/commit/b2886e64b4aba660e4939d6cdaba066aed39d081
>>> 
>>> Since we have divided opinions on this issue, would you mind reviewing the discussion in the comments and add your own thoughts?
>>> 
>>> Best regards
>>> /Andreas
>>> 
>>> 
>> 
>> 
>> I don?t really see the difference because I don?t know the code. If
>> the APIs enable me, or anyone who doesn?t know the text code to do
>> stuff like insert text at a certain point for example, the it would
>> seem ok.
>> One thing that is mentioned is for example inserting a QChar.. so how
>> does one programmatically insert 1 character?
> 
> StoryText has no API to insert single character, inserting QChar
> apparently works because it can be cast to QString or some such. You can
> now either insert a QString and the code will convert it to UTF-32, or a
> QVector<uint> of UTF-32 code points. We might have a friendly typedef
> for this vector.
> 
>> I don?t have an opinion on the 16 vs 32 as long as it works so we can
>> do all operations.. read/write to file, display it, type it, export it
>> to various formats.
> 
> The big difference is that UTF-16 is variable length, a Unicode
> character can be represented by one or two UTF-16 code units, but
> Scribus was treating it as a fixed length encoding which is wrong.
> UTF-32, on the other hand, is fixed length encoding, so the assumption
> all over Scribus codebase can be kept.
> 

So? you don?t want to use UTF16 as its a variable length encoding (1 or 2 16 bit code units) and we?d need to deal with the unknown length in all the code..

And the downside to UTF32 is? Higher memory requirement? Inability to directly use QStrings in storage of text? (even though there is QString::toUcs4 to facilitate conversion)

http://unicode.org/faq/utf_bom.html#utf32-2:
?
Q: Should I use UTF-32 (or UCS-4) for storing Unicode strings in memory?

A: This depends. If you frequently need to access APIs that require string parameters to be in UTF-32, it may be more convenient to work with UTF-32 strings all the time. However, the downside of UTF-32 is that it forces you to use 32-bits for each character, when only 21 bits are ever needed. The number of significant bits needed for the average character in common texts is much lower, making the ratio effectively that much worse. In many situations that does not matter, and the convenience of having a fixed number of code units per character can be the deciding factor.

Increasing the storage for the same number of characters does have its cost in applications dealing with large volume of text data: it can mean exhausting cache limits sooner; it can result in noticeably increased read/write times or in reaching bandwidth limits; and it requires more space for storage. What a number of implementations do is to represent strings with UTF-16,	but individual character values with UTF-32.

The chief selling point for Unicode is providing a representation for all the world?s characters, eliminating the need for juggling multiple character sets and avoiding the associated data corruption problems. These features were enough to swing industry to the side of using Unicode (UTF-16). While a UTF-32 representation does make the programming model somewhat simpler, the increased average storage size has real drawbacks, making a complete transition to UTF-32 less compelling.
?

Jean?

Avox: why not UTF32 in your opinion?

Craig







From khaledhosny at eglug.org  Mon May 30 19:36:34 2016
From: khaledhosny at eglug.org (Khaled Hosny)
Date: Mon, 30 May 2016 21:36:34 +0200
Subject: [scribus-dev] Change to UTF-32
In-Reply-To: <FC43BF8B-9CD6-438E-94B0-DDC2631D0941@scribus.info>
References: <016001d1ba87$b2ee0670$18ca1350$@googlemail.com>
 <3E21CCFD-A756-4D47-982D-4C691029AA21@scribus.info>
 <20160530191428.GA20467@macbook>
 <FC43BF8B-9CD6-438E-94B0-DDC2631D0941@scribus.info>
Message-ID: <20160530193634.GB20467@macbook>

On Mon, May 30, 2016 at 09:22:43PM +0200, Craig Bradney wrote:
> 
> > On 30 May 2016, at 21:14, Khaled Hosny <khaledhosny at eglug.org> wrote:
> > 
> > On Mon, May 30, 2016 at 08:56:06PM +0200, Craig Bradney wrote:
> >> 
> >>> On 30 May 2016, at 17:26, Andre?a?s? <andreas.vox at gmail.com> wrote:
> >>> 
> >>> Hi!
> >>> 
> >>> Khaled recently committed a change to the HOST-Oman/ctl branch that changes the text coding from UTF_16 (QChar/QString) to UTF-32:
> >>> https://github.com/HOST-Oman/scribus/commit/b2886e64b4aba660e4939d6cdaba066aed39d081
> >>> 
> >>> Since we have divided opinions on this issue, would you mind reviewing the discussion in the comments and add your own thoughts?
> >>> 
> >>> Best regards
> >>> /Andreas
> >>> 
> >>> 
> >> 
> >> 
> >> I don?t really see the difference because I don?t know the code. If
> >> the APIs enable me, or anyone who doesn?t know the text code to do
> >> stuff like insert text at a certain point for example, the it would
> >> seem ok.
> >> One thing that is mentioned is for example inserting a QChar.. so how
> >> does one programmatically insert 1 character?
> > 
> > StoryText has no API to insert single character, inserting QChar
> > apparently works because it can be cast to QString or some such. You can
> > now either insert a QString and the code will convert it to UTF-32, or a
> > QVector<uint> of UTF-32 code points. We might have a friendly typedef
> > for this vector.
> > 
> >> I don?t have an opinion on the 16 vs 32 as long as it works so we can
> >> do all operations.. read/write to file, display it, type it, export it
> >> to various formats.
> > 
> > The big difference is that UTF-16 is variable length, a Unicode
> > character can be represented by one or two UTF-16 code units, but
> > Scribus was treating it as a fixed length encoding which is wrong.
> > UTF-32, on the other hand, is fixed length encoding, so the assumption
> > all over Scribus codebase can be kept.
> > 
> 
> So? you don?t want to use UTF16 as its a variable length encoding (1
> or 2 16 bit code units) and we?d need to deal with the unknown length
> in all the code..

Right.

> And the downside to UTF32 is? Higher memory requirement?

Not sure about that, since we are not replacing 16-bit int by 32-bit
int, but QChar by uint, whether uint takes less or more memory than
QChar this needs to be measured (no idea how, though).

> Inability to
> directly use QStrings in storage of text? (even though there is
> QString::toUcs4 to facilitate conversion)

We can (and still) use QString in many places, mainly when interfacing
with APIs that use QString. We just need to be careful and not assume
that QString length is the same as StoryText length.

Regards,
Khaled


From khaledhosny at eglug.org  Mon May 30 19:38:15 2016
From: khaledhosny at eglug.org (Khaled Hosny)
Date: Mon, 30 May 2016 21:38:15 +0200
Subject: [scribus-dev] Change to UTF-32
In-Reply-To: <CAM9PYkaqzObPk7NVmcr=kAMMv7X3Di4oMUS2kaJPmBPt4BPJ_w@mail.gmail.com>
References: <016001d1ba87$b2ee0670$18ca1350$@googlemail.com>
 <3E21CCFD-A756-4D47-982D-4C691029AA21@scribus.info>
 <CAM9PYkaqzObPk7NVmcr=kAMMv7X3Di4oMUS2kaJPmBPt4BPJ_w@mail.gmail.com>
Message-ID: <20160530193815.GC20467@macbook>

On Mon, May 30, 2016 at 03:01:58PM -0400, Louis Desjardins wrote:
> 2016-05-30 14:56 GMT-04:00 Craig Bradney <cbradney at scribus.info>:
> 
> >
> > > On 30 May 2016, at 17:26, Andre?a?s? <andreas.vox at gmail.com> wrote:
> > >
> > > Hi!
> > >
> > > Khaled recently committed a change to the HOST-Oman/ctl branch that
> > changes the text coding from UTF_16 (QChar/QString) to UTF-32:
> > >
> > https://github.com/HOST-Oman/scribus/commit/b2886e64b4aba660e4939d6cdaba066aed39d081
> > >
> > > Since we have divided opinions on this issue, would you mind reviewing
> > the discussion in the comments and add your own thoughts?
> > >
> > > Best regards
> > > /Andreas
> > >
> > >
> >
> >
> > I don?t really see the difference because I don?t know the code. If the
> > APIs enable me, or anyone who doesn?t know the text code to do stuff like
> > insert text at a certain point for example, the it would seem ok.
> > One thing that is mentioned is for example inserting a QChar.. so how does
> > one programmatically insert 1 character?
> >
> > I don?t have an opinion on the 16 vs 32 as long as it works so we can do
> > all operations.. read/write to file, display it, type it, export it to
> > various formats.
> >
> 
> Does this matter at text importation?

The only difference is that Scribus can now import and export text
outside Basic Multilingual Plane correctly.

Regards,
Khaled


From cbradney at scribus.info  Mon May 30 19:39:11 2016
From: cbradney at scribus.info (Craig Bradney)
Date: Mon, 30 May 2016 21:39:11 +0200
Subject: [scribus-dev] Change to UTF-32
In-Reply-To: <20160530193634.GB20467@macbook>
References: <016001d1ba87$b2ee0670$18ca1350$@googlemail.com>
 <3E21CCFD-A756-4D47-982D-4C691029AA21@scribus.info>
 <20160530191428.GA20467@macbook>
 <FC43BF8B-9CD6-438E-94B0-DDC2631D0941@scribus.info>
 <20160530193634.GB20467@macbook>
Message-ID: <E176536F-ECA3-40B7-8CB0-03436B09F72E@scribus.info>


> On 30 May 2016, at 21:36, Khaled Hosny <khaledhosny at eglug.org> wrote:
> 
> On Mon, May 30, 2016 at 09:22:43PM +0200, Craig Bradney wrote:
>> 
>>> On 30 May 2016, at 21:14, Khaled Hosny <khaledhosny at eglug.org> wrote:
>>> 
>>> On Mon, May 30, 2016 at 08:56:06PM +0200, Craig Bradney wrote:
>>>> 
>>>>> On 30 May 2016, at 17:26, Andre?a?s? <andreas.vox at gmail.com> wrote:
>>>>> 
>>>>> Hi!
>>>>> 
>>>>> Khaled recently committed a change to the HOST-Oman/ctl branch that changes the text coding from UTF_16 (QChar/QString) to UTF-32:
>>>>> https://github.com/HOST-Oman/scribus/commit/b2886e64b4aba660e4939d6cdaba066aed39d081
>>>>> 
>>>>> Since we have divided opinions on this issue, would you mind reviewing the discussion in the comments and add your own thoughts?
>>>>> 
>>>>> Best regards
>>>>> /Andreas
>>>>> 
>>>>> 
>>>> 
>>>> 
>>>> I don?t really see the difference because I don?t know the code. If
>>>> the APIs enable me, or anyone who doesn?t know the text code to do
>>>> stuff like insert text at a certain point for example, the it would
>>>> seem ok.
>>>> One thing that is mentioned is for example inserting a QChar.. so how
>>>> does one programmatically insert 1 character?
>>> 
>>> StoryText has no API to insert single character, inserting QChar
>>> apparently works because it can be cast to QString or some such. You can
>>> now either insert a QString and the code will convert it to UTF-32, or a
>>> QVector<uint> of UTF-32 code points. We might have a friendly typedef
>>> for this vector.
>>> 
>>>> I don?t have an opinion on the 16 vs 32 as long as it works so we can
>>>> do all operations.. read/write to file, display it, type it, export it
>>>> to various formats.
>>> 
>>> The big difference is that UTF-16 is variable length, a Unicode
>>> character can be represented by one or two UTF-16 code units, but
>>> Scribus was treating it as a fixed length encoding which is wrong.
>>> UTF-32, on the other hand, is fixed length encoding, so the assumption
>>> all over Scribus codebase can be kept.
>>> 
>> 
>> So? you don?t want to use UTF16 as its a variable length encoding (1
>> or 2 16 bit code units) and we?d need to deal with the unknown length
>> in all the code..
> 
> Right.
> 
>> And the downside to UTF32 is? Higher memory requirement?
> 
> Not sure about that, since we are not replacing 16-bit int by 32-bit
> int, but QChar by uint, whether uint takes less or more memory than
> QChar this needs to be measured (no idea how, though).
> 
>> Inability to
>> directly use QStrings in storage of text? (even though there is
>> QString::toUcs4 to facilitate conversion)
> 
> We can (and still) use QString in many places, mainly when interfacing
> with APIs that use QString. We just need to be careful and not assume
> that QString length is the same as StoryText length.
> 

What about speed? We already need to speed up the text engine massively, including rendering (or especially rendering).

Craig



From khaledhosny at eglug.org  Mon May 30 20:08:24 2016
From: khaledhosny at eglug.org (Khaled Hosny)
Date: Mon, 30 May 2016 22:08:24 +0200
Subject: [scribus-dev] Change to UTF-32
In-Reply-To: <E176536F-ECA3-40B7-8CB0-03436B09F72E@scribus.info>
References: <016001d1ba87$b2ee0670$18ca1350$@googlemail.com>
 <3E21CCFD-A756-4D47-982D-4C691029AA21@scribus.info>
 <20160530191428.GA20467@macbook>
 <FC43BF8B-9CD6-438E-94B0-DDC2631D0941@scribus.info>
 <20160530193634.GB20467@macbook>
 <E176536F-ECA3-40B7-8CB0-03436B09F72E@scribus.info>
Message-ID: <20160530200824.GD20467@macbook>

On Mon, May 30, 2016 at 09:39:11PM +0200, Craig Bradney wrote:
> 
> > On 30 May 2016, at 21:36, Khaled Hosny <khaledhosny at eglug.org> wrote:
> > 
> > On Mon, May 30, 2016 at 09:22:43PM +0200, Craig Bradney wrote:
> >> 
> >>> On 30 May 2016, at 21:14, Khaled Hosny <khaledhosny at eglug.org> wrote:
> >>> 
> >>> On Mon, May 30, 2016 at 08:56:06PM +0200, Craig Bradney wrote:
> >>>> 
> >>>>> On 30 May 2016, at 17:26, Andre?a?s? <andreas.vox at gmail.com> wrote:
> >>>>> 
> >>>>> Hi!
> >>>>> 
> >>>>> Khaled recently committed a change to the HOST-Oman/ctl branch that changes the text coding from UTF_16 (QChar/QString) to UTF-32:
> >>>>> https://github.com/HOST-Oman/scribus/commit/b2886e64b4aba660e4939d6cdaba066aed39d081
> >>>>> 
> >>>>> Since we have divided opinions on this issue, would you mind reviewing the discussion in the comments and add your own thoughts?
> >>>>> 
> >>>>> Best regards
> >>>>> /Andreas
> >>>>> 
> >>>>> 
> >>>> 
> >>>> 
> >>>> I don?t really see the difference because I don?t know the code. If
> >>>> the APIs enable me, or anyone who doesn?t know the text code to do
> >>>> stuff like insert text at a certain point for example, the it would
> >>>> seem ok.
> >>>> One thing that is mentioned is for example inserting a QChar.. so how
> >>>> does one programmatically insert 1 character?
> >>> 
> >>> StoryText has no API to insert single character, inserting QChar
> >>> apparently works because it can be cast to QString or some such. You can
> >>> now either insert a QString and the code will convert it to UTF-32, or a
> >>> QVector<uint> of UTF-32 code points. We might have a friendly typedef
> >>> for this vector.
> >>> 
> >>>> I don?t have an opinion on the 16 vs 32 as long as it works so we can
> >>>> do all operations.. read/write to file, display it, type it, export it
> >>>> to various formats.
> >>> 
> >>> The big difference is that UTF-16 is variable length, a Unicode
> >>> character can be represented by one or two UTF-16 code units, but
> >>> Scribus was treating it as a fixed length encoding which is wrong.
> >>> UTF-32, on the other hand, is fixed length encoding, so the assumption
> >>> all over Scribus codebase can be kept.
> >>> 
> >> 
> >> So? you don?t want to use UTF16 as its a variable length encoding (1
> >> or 2 16 bit code units) and we?d need to deal with the unknown length
> >> in all the code..
> > 
> > Right.
> > 
> >> And the downside to UTF32 is? Higher memory requirement?
> > 
> > Not sure about that, since we are not replacing 16-bit int by 32-bit
> > int, but QChar by uint, whether uint takes less or more memory than
> > QChar this needs to be measured (no idea how, though).
> > 
> >> Inability to
> >> directly use QStrings in storage of text? (even though there is
> >> QString::toUcs4 to facilitate conversion)
> > 
> > We can (and still) use QString in many places, mainly when interfacing
> > with APIs that use QString. We just need to be careful and not assume
> > that QString length is the same as StoryText length.
> > 
> 
> What about speed? We already need to speed up the text engine massively, including rendering (or especially rendering).

Converting a QString read from 25MB text file to UTF-32 takes <
0.4?seconds, so that does not seem to be a concern. The current slowness
is mainly bad handling of text spanning multiple frames more than
anything else and Andreas is working on a fix for this.




From cbradney at scribus.info  Mon May 30 20:10:08 2016
From: cbradney at scribus.info (Craig Bradney)
Date: Mon, 30 May 2016 22:10:08 +0200
Subject: [scribus-dev] Change to UTF-32
In-Reply-To: <20160530200824.GD20467@macbook>
References: <016001d1ba87$b2ee0670$18ca1350$@googlemail.com>
 <3E21CCFD-A756-4D47-982D-4C691029AA21@scribus.info>
 <20160530191428.GA20467@macbook>
 <FC43BF8B-9CD6-438E-94B0-DDC2631D0941@scribus.info>
 <20160530193634.GB20467@macbook>
 <E176536F-ECA3-40B7-8CB0-03436B09F72E@scribus.info>
 <20160530200824.GD20467@macbook>
Message-ID: <05A5F268-151B-40C0-8378-5997C3D5DA8A@scribus.info>


> On 30 May 2016, at 22:08, Khaled Hosny <khaledhosny at eglug.org> wrote:
> 
> On Mon, May 30, 2016 at 09:39:11PM +0200, Craig Bradney wrote:
>> 
>>> On 30 May 2016, at 21:36, Khaled Hosny <khaledhosny at eglug.org> wrote:
>>> 
>>> On Mon, May 30, 2016 at 09:22:43PM +0200, Craig Bradney wrote:
>>>> 
>>>>> On 30 May 2016, at 21:14, Khaled Hosny <khaledhosny at eglug.org> wrote:
>>>>> 
>>>>> On Mon, May 30, 2016 at 08:56:06PM +0200, Craig Bradney wrote:
>>>>>> 
>>>>>>> On 30 May 2016, at 17:26, Andre?a?s? <andreas.vox at gmail.com> wrote:
>>>>>>> 
>>>>>>> Hi!
>>>>>>> 
>>>>>>> Khaled recently committed a change to the HOST-Oman/ctl branch that changes the text coding from UTF_16 (QChar/QString) to UTF-32:
>>>>>>> https://github.com/HOST-Oman/scribus/commit/b2886e64b4aba660e4939d6cdaba066aed39d081
>>>>>>> 
>>>>>>> Since we have divided opinions on this issue, would you mind reviewing the discussion in the comments and add your own thoughts?
>>>>>>> 
>>>>>>> Best regards
>>>>>>> /Andreas
>>>>>>> 
>>>>>>> 
>>>>>> 
>>>>>> 
>>>>>> I don?t really see the difference because I don?t know the code. If
>>>>>> the APIs enable me, or anyone who doesn?t know the text code to do
>>>>>> stuff like insert text at a certain point for example, the it would
>>>>>> seem ok.
>>>>>> One thing that is mentioned is for example inserting a QChar.. so how
>>>>>> does one programmatically insert 1 character?
>>>>> 
>>>>> StoryText has no API to insert single character, inserting QChar
>>>>> apparently works because it can be cast to QString or some such. You can
>>>>> now either insert a QString and the code will convert it to UTF-32, or a
>>>>> QVector<uint> of UTF-32 code points. We might have a friendly typedef
>>>>> for this vector.
>>>>> 
>>>>>> I don?t have an opinion on the 16 vs 32 as long as it works so we can
>>>>>> do all operations.. read/write to file, display it, type it, export it
>>>>>> to various formats.
>>>>> 
>>>>> The big difference is that UTF-16 is variable length, a Unicode
>>>>> character can be represented by one or two UTF-16 code units, but
>>>>> Scribus was treating it as a fixed length encoding which is wrong.
>>>>> UTF-32, on the other hand, is fixed length encoding, so the assumption
>>>>> all over Scribus codebase can be kept.
>>>>> 
>>>> 
>>>> So? you don?t want to use UTF16 as its a variable length encoding (1
>>>> or 2 16 bit code units) and we?d need to deal with the unknown length
>>>> in all the code..
>>> 
>>> Right.
>>> 
>>>> And the downside to UTF32 is? Higher memory requirement?
>>> 
>>> Not sure about that, since we are not replacing 16-bit int by 32-bit
>>> int, but QChar by uint, whether uint takes less or more memory than
>>> QChar this needs to be measured (no idea how, though).
>>> 
>>>> Inability to
>>>> directly use QStrings in storage of text? (even though there is
>>>> QString::toUcs4 to facilitate conversion)
>>> 
>>> We can (and still) use QString in many places, mainly when interfacing
>>> with APIs that use QString. We just need to be careful and not assume
>>> that QString length is the same as StoryText length.
>>> 
>> 
>> What about speed? We already need to speed up the text engine massively, including rendering (or especially rendering).
> 
> Converting a QString read from 25MB text file to UTF-32 takes <
> 0.4 seconds, so that does not seem to be a concern. The current slowness
> is mainly bad handling of text spanning multiple frames more than
> anything else and Andreas is working on a fix for this.
> 
> 


http://unicode.org/faq/utf_bom.html#UTF32 has some fairly strong wording about slow down from using UTF32.


Craig



From gpittman at iglou.com  Mon May 30 20:14:42 2016
From: gpittman at iglou.com (Gregory Pittman)
Date: Mon, 30 May 2016 16:14:42 -0400
Subject: [scribus-dev] Change to UTF-32
In-Reply-To: <E176536F-ECA3-40B7-8CB0-03436B09F72E@scribus.info>
References: <016001d1ba87$b2ee0670$18ca1350$@googlemail.com>
 <3E21CCFD-A756-4D47-982D-4C691029AA21@scribus.info>
 <20160530191428.GA20467@macbook>
 <FC43BF8B-9CD6-438E-94B0-DDC2631D0941@scribus.info>
 <20160530193634.GB20467@macbook>
 <E176536F-ECA3-40B7-8CB0-03436B09F72E@scribus.info>
Message-ID: <76b86124-6535-cb3b-04c9-90853109e730@iglou.com>

On 05/30/2016 03:39 PM, Craig Bradney wrote:

> 
> What about speed? We already need to speed up the text engine massively, including rendering (or especially rendering).
> 

As far as Scripter, one place to look for trouble might be with
Autoquote/Autoquote2, though my hunch is that these will work as expected.

Greg


From khaledhosny at eglug.org  Mon May 30 20:24:37 2016
From: khaledhosny at eglug.org (Khaled Hosny)
Date: Mon, 30 May 2016 22:24:37 +0200
Subject: [scribus-dev] Change to UTF-32
In-Reply-To: <05A5F268-151B-40C0-8378-5997C3D5DA8A@scribus.info>
References: <016001d1ba87$b2ee0670$18ca1350$@googlemail.com>
 <3E21CCFD-A756-4D47-982D-4C691029AA21@scribus.info>
 <20160530191428.GA20467@macbook>
 <FC43BF8B-9CD6-438E-94B0-DDC2631D0941@scribus.info>
 <20160530193634.GB20467@macbook>
 <E176536F-ECA3-40B7-8CB0-03436B09F72E@scribus.info>
 <20160530200824.GD20467@macbook>
 <05A5F268-151B-40C0-8378-5997C3D5DA8A@scribus.info>
Message-ID: <20160530202437.GE20467@macbook>

On Mon, May 30, 2016 at 10:10:08PM +0200, Craig Bradney wrote:
> 
> > On 30 May 2016, at 22:08, Khaled Hosny <khaledhosny at eglug.org> wrote:
> > 
> > On Mon, May 30, 2016 at 09:39:11PM +0200, Craig Bradney wrote:
> >> 
> >>> On 30 May 2016, at 21:36, Khaled Hosny <khaledhosny at eglug.org> wrote:
> >>> 
> >>> On Mon, May 30, 2016 at 09:22:43PM +0200, Craig Bradney wrote:
> >>>> 
> >>>>> On 30 May 2016, at 21:14, Khaled Hosny <khaledhosny at eglug.org> wrote:
> >>>>> 
> >>>>> On Mon, May 30, 2016 at 08:56:06PM +0200, Craig Bradney wrote:
> >>>>>> 
> >>>>>>> On 30 May 2016, at 17:26, Andre?a?s? <andreas.vox at gmail.com> wrote:
> >>>>>>> 
> >>>>>>> Hi!
> >>>>>>> 
> >>>>>>> Khaled recently committed a change to the HOST-Oman/ctl branch that changes the text coding from UTF_16 (QChar/QString) to UTF-32:
> >>>>>>> https://github.com/HOST-Oman/scribus/commit/b2886e64b4aba660e4939d6cdaba066aed39d081
> >>>>>>> 
> >>>>>>> Since we have divided opinions on this issue, would you mind reviewing the discussion in the comments and add your own thoughts?
> >>>>>>> 
> >>>>>>> Best regards
> >>>>>>> /Andreas
> >>>>>>> 
> >>>>>>> 
> >>>>>> 
> >>>>>> 
> >>>>>> I don?t really see the difference because I don?t know the code. If
> >>>>>> the APIs enable me, or anyone who doesn?t know the text code to do
> >>>>>> stuff like insert text at a certain point for example, the it would
> >>>>>> seem ok.
> >>>>>> One thing that is mentioned is for example inserting a QChar.. so how
> >>>>>> does one programmatically insert 1 character?
> >>>>> 
> >>>>> StoryText has no API to insert single character, inserting QChar
> >>>>> apparently works because it can be cast to QString or some such. You can
> >>>>> now either insert a QString and the code will convert it to UTF-32, or a
> >>>>> QVector<uint> of UTF-32 code points. We might have a friendly typedef
> >>>>> for this vector.
> >>>>> 
> >>>>>> I don?t have an opinion on the 16 vs 32 as long as it works so we can
> >>>>>> do all operations.. read/write to file, display it, type it, export it
> >>>>>> to various formats.
> >>>>> 
> >>>>> The big difference is that UTF-16 is variable length, a Unicode
> >>>>> character can be represented by one or two UTF-16 code units, but
> >>>>> Scribus was treating it as a fixed length encoding which is wrong.
> >>>>> UTF-32, on the other hand, is fixed length encoding, so the assumption
> >>>>> all over Scribus codebase can be kept.
> >>>>> 
> >>>> 
> >>>> So? you don?t want to use UTF16 as its a variable length encoding (1
> >>>> or 2 16 bit code units) and we?d need to deal with the unknown length
> >>>> in all the code..
> >>> 
> >>> Right.
> >>> 
> >>>> And the downside to UTF32 is? Higher memory requirement?
> >>> 
> >>> Not sure about that, since we are not replacing 16-bit int by 32-bit
> >>> int, but QChar by uint, whether uint takes less or more memory than
> >>> QChar this needs to be measured (no idea how, though).
> >>> 
> >>>> Inability to
> >>>> directly use QStrings in storage of text? (even though there is
> >>>> QString::toUcs4 to facilitate conversion)
> >>> 
> >>> We can (and still) use QString in many places, mainly when interfacing
> >>> with APIs that use QString. We just need to be careful and not assume
> >>> that QString length is the same as StoryText length.
> >>> 
> >> 
> >> What about speed? We already need to speed up the text engine massively, including rendering (or especially rendering).
> > 
> > Converting a QString read from 25MB text file to UTF-32 takes <
> > 0.4 seconds, so that does not seem to be a concern. The current slowness
> > is mainly bad handling of text spanning multiple frames more than
> > anything else and Andreas is working on a fix for this.
> > 
> > 
> 
> http://unicode.org/faq/utf_bom.html#UTF32 has some fairly strong wording about slow down from using UTF32.

That seems like red herring mostly, if it were that bad we should have
been using UTF-8 from the start. A 1 million character string takes ~2MB
of memory as UTF-16 and a whopping ~4MB as UTF-32. I fail to see what
kind of slowing this might give us in 2016. Either this page is talking
about processing hundreds of gigabytes of text (not something Scribus
does) or computers from the last century.


From louis.desjardins at gmail.com  Mon May 30 20:29:51 2016
From: louis.desjardins at gmail.com (Louis Desjardins)
Date: Mon, 30 May 2016 16:29:51 -0400
Subject: [scribus-dev] Change to UTF-32
In-Reply-To: <20160530202437.GE20467@macbook>
References: <016001d1ba87$b2ee0670$18ca1350$@googlemail.com>
 <3E21CCFD-A756-4D47-982D-4C691029AA21@scribus.info>
 <20160530191428.GA20467@macbook>
 <FC43BF8B-9CD6-438E-94B0-DDC2631D0941@scribus.info>
 <20160530193634.GB20467@macbook>
 <E176536F-ECA3-40B7-8CB0-03436B09F72E@scribus.info>
 <20160530200824.GD20467@macbook>
 <05A5F268-151B-40C0-8378-5997C3D5DA8A@scribus.info>
 <20160530202437.GE20467@macbook>
Message-ID: <CAM9PYkaZROVUDa=BFpN+ROTchKjscfmJqmUcHYu+Bn2JZHFnfw@mail.gmail.com>

2016-05-30 16:24 GMT-04:00 Khaled Hosny <khaledhosny at eglug.org>:

> On Mon, May 30, 2016 at 10:10:08PM +0200, Craig Bradney wrote:
> >
> > > On 30 May 2016, at 22:08, Khaled Hosny <khaledhosny at eglug.org> wrote:
> > >
> > > On Mon, May 30, 2016 at 09:39:11PM +0200, Craig Bradney wrote:
> > >>
> > >>> On 30 May 2016, at 21:36, Khaled Hosny <khaledhosny at eglug.org>
> wrote:
> > >>>
> > >>> On Mon, May 30, 2016 at 09:22:43PM +0200, Craig Bradney wrote:
> > >>>>
> > >>>>> On 30 May 2016, at 21:14, Khaled Hosny <khaledhosny at eglug.org>
> wrote:
> > >>>>>
> > >>>>> On Mon, May 30, 2016 at 08:56:06PM +0200, Craig Bradney wrote:
> > >>>>>>
> > >>>>>>> On 30 May 2016, at 17:26, Andre?a?s? <andreas.vox at gmail.com>
> wrote:
> > >>>>>>>
> > >>>>>>> Hi!
> > >>>>>>>
> > >>>>>>> Khaled recently committed a change to the HOST-Oman/ctl branch
> that changes the text coding from UTF_16 (QChar/QString) to UTF-32:
> > >>>>>>>
> https://github.com/HOST-Oman/scribus/commit/b2886e64b4aba660e4939d6cdaba066aed39d081
> > >>>>>>>
> > >>>>>>> Since we have divided opinions on this issue, would you mind
> reviewing the discussion in the comments and add your own thoughts?
> > >>>>>>>
> > >>>>>>> Best regards
> > >>>>>>> /Andreas
> > >>>>>>>
> > >>>>>>>
> > >>>>>>
> > >>>>>>
> > >>>>>> I don?t really see the difference because I don?t know the code.
> If
> > >>>>>> the APIs enable me, or anyone who doesn?t know the text code to do
> > >>>>>> stuff like insert text at a certain point for example, the it
> would
> > >>>>>> seem ok.
> > >>>>>> One thing that is mentioned is for example inserting a QChar.. so
> how
> > >>>>>> does one programmatically insert 1 character?
> > >>>>>
> > >>>>> StoryText has no API to insert single character, inserting QChar
> > >>>>> apparently works because it can be cast to QString or some such.
> You can
> > >>>>> now either insert a QString and the code will convert it to
> UTF-32, or a
> > >>>>> QVector<uint> of UTF-32 code points. We might have a friendly
> typedef
> > >>>>> for this vector.
> > >>>>>
> > >>>>>> I don?t have an opinion on the 16 vs 32 as long as it works so we
> can
> > >>>>>> do all operations.. read/write to file, display it, type it,
> export it
> > >>>>>> to various formats.
> > >>>>>
> > >>>>> The big difference is that UTF-16 is variable length, a Unicode
> > >>>>> character can be represented by one or two UTF-16 code units, but
> > >>>>> Scribus was treating it as a fixed length encoding which is wrong.
> > >>>>> UTF-32, on the other hand, is fixed length encoding, so the
> assumption
> > >>>>> all over Scribus codebase can be kept.
> > >>>>>
> > >>>>
> > >>>> So? you don?t want to use UTF16 as its a variable length encoding (1
> > >>>> or 2 16 bit code units) and we?d need to deal with the unknown
> length
> > >>>> in all the code..
> > >>>
> > >>> Right.
> > >>>
> > >>>> And the downside to UTF32 is? Higher memory requirement?
> > >>>
> > >>> Not sure about that, since we are not replacing 16-bit int by 32-bit
> > >>> int, but QChar by uint, whether uint takes less or more memory than
> > >>> QChar this needs to be measured (no idea how, though).
> > >>>
> > >>>> Inability to
> > >>>> directly use QStrings in storage of text? (even though there is
> > >>>> QString::toUcs4 to facilitate conversion)
> > >>>
> > >>> We can (and still) use QString in many places, mainly when
> interfacing
> > >>> with APIs that use QString. We just need to be careful and not assume
> > >>> that QString length is the same as StoryText length.
> > >>>
> > >>
> > >> What about speed? We already need to speed up the text engine
> massively, including rendering (or especially rendering).
> > >
> > > Converting a QString read from 25MB text file to UTF-32 takes <
> > > 0.4 seconds, so that does not seem to be a concern. The current
> slowness
> > > is mainly bad handling of text spanning multiple frames more than
> > > anything else and Andreas is working on a fix for this.
> > >
> > >
> >
> > http://unicode.org/faq/utf_bom.html#UTF32 has some fairly strong
> wording about slow down from using UTF32.
>
> That seems like red herring mostly, if it were that bad we should have
> been using UTF-8 from the start. A 1 million character string takes ~2MB
> of memory as UTF-16 and a whopping ~4MB as UTF-32. I fail to see what
> kind of slowing this might give us in 2016. Either this page is talking
> about processing hundreds of gigabytes of text (not something Scribus
> does) or computers from the last century.
>

Correct me if I?m wrong but that is maybe why there used to be 2 versions,
one Latin and one CJK, for the commercial DTP apps. That?s now history.
Back then, the computer were, as you point out, much (much) slower, hence
the necessity of this kind of solution.

Louis

>
> _______________________________________________
> scribus-dev mailing list
> scribus-dev at lists.scribus.net
> http://lists.scribus.net/mailman/listinfo/scribus-dev
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.scribus.net/pipermail/scribus-dev/attachments/20160530/254fa2c5/attachment.html>

From andreas.vox at gmail.com  Mon May 30 22:41:25 2016
From: andreas.vox at gmail.com (Andreas Vox)
Date: Tue, 31 May 2016 00:41:25 +0200
Subject: [scribus-dev] Change to UTF-32
In-Reply-To: <05A5F268-151B-40C0-8378-5997C3D5DA8A@scribus.info>
References: <016001d1ba87$b2ee0670$18ca1350$@googlemail.com>
 <3E21CCFD-A756-4D47-982D-4C691029AA21@scribus.info>
 <20160530191428.GA20467@macbook>
 <FC43BF8B-9CD6-438E-94B0-DDC2631D0941@scribus.info>
 <20160530193634.GB20467@macbook>
 <E176536F-ECA3-40B7-8CB0-03436B09F72E@scribus.info>
 <20160530200824.GD20467@macbook>
 <05A5F268-151B-40C0-8378-5997C3D5DA8A@scribus.info>
Message-ID: <4C82221D-A19B-4DF9-9A2A-2D023E90A9DE@gmail.com>


> Am 30.05.2016 um 22:10 schrieb Craig Bradney <cbradney at scribus.info>:
> 
> 
>> On 30 May 2016, at 22:08, Khaled Hosny <khaledhosny at eglug.org> wrote:
>> 
>> On Mon, May 30, 2016 at 09:39:11PM +0200, Craig Bradney wrote:
>>> 
>>>> 
>>> 
>>> What about speed? We already need to speed up the text engine massively, including rendering (or especially rendering).
>> 
>> Converting a QString read from 25MB text file to UTF-32 takes <
>> 0.4 seconds, so that does not seem to be a concern. The current slowness
>> is mainly bad handling of text spanning multiple frames more than
>> anything else and Andreas is working on a fix for this.
>> 
>> 
> 
> 
> http://unicode.org/faq/utf_bom.html#UTF32 has some fairly strong wording about slow down from using UTF32.
> 

I wonder what they?d say if they saw our code in ScText_Shared?

As long as we pack every Char in an ScText struct, 8 vs. 16 vs. 32 bit doesn?t matter. A better structure would use a rope data structure or a list of strings for text and store style information as runs outside the text (coming soon).

My main issue is that we won?t be using QString as a model for story text. String has many handy member functions that QLsit<uint> doesn?t. It?s also possible that QList<uint> is slower for many tasks.

But if everyone is fine with the change, we can make it work. Text search and nextCharacter() have to be rewritten anyway in order to handle non-latin text correctly.

The file format will probably use UTF8 anyway. Importing/exporting text will always handle character encoding as needed.

/Andreas

From cbradney at scribus.info  Tue May 31 05:11:47 2016
From: cbradney at scribus.info (Craig Bradney)
Date: Tue, 31 May 2016 07:11:47 +0200
Subject: [scribus-dev] Change to UTF-32
In-Reply-To: <4C82221D-A19B-4DF9-9A2A-2D023E90A9DE@gmail.com>
References: <016001d1ba87$b2ee0670$18ca1350$@googlemail.com>
 <3E21CCFD-A756-4D47-982D-4C691029AA21@scribus.info>
 <20160530191428.GA20467@macbook>
 <FC43BF8B-9CD6-438E-94B0-DDC2631D0941@scribus.info>
 <20160530193634.GB20467@macbook>
 <E176536F-ECA3-40B7-8CB0-03436B09F72E@scribus.info>
 <20160530200824.GD20467@macbook>
 <05A5F268-151B-40C0-8378-5997C3D5DA8A@scribus.info>
 <4C82221D-A19B-4DF9-9A2A-2D023E90A9DE@gmail.com>
Message-ID: <2ADBFA59-CE7D-4EA0-BC60-C7E67CB3EAC3@scribus.info>


> On 31 May 2016, at 00:41, Andreas Vox <andreas.vox at gmail.com> wrote:
> 
> 
>> Am 30.05.2016 um 22:10 schrieb Craig Bradney <cbradney at scribus.info>:
>> 
>> 
>>> On 30 May 2016, at 22:08, Khaled Hosny <khaledhosny at eglug.org> wrote:
>>> 
>>> On Mon, May 30, 2016 at 09:39:11PM +0200, Craig Bradney wrote:
>>>> 
>>>>> 
>>>> 
>>>> What about speed? We already need to speed up the text engine massively, including rendering (or especially rendering).
>>> 
>>> Converting a QString read from 25MB text file to UTF-32 takes <
>>> 0.4 seconds, so that does not seem to be a concern. The current slowness
>>> is mainly bad handling of text spanning multiple frames more than
>>> anything else and Andreas is working on a fix for this.
>>> 
>>> 
>> 
>> 
>> http://unicode.org/faq/utf_bom.html#UTF32 has some fairly strong wording about slow down from using UTF32.
>> 
> 
> I wonder what they?d say if they saw our code in ScText_Shared?
> 
> As long as we pack every Char in an ScText struct, 8 vs. 16 vs. 32 bit doesn?t matter. A better structure would use a rope data structure or a list of strings for text and store style information as runs outside the text (coming soon).
> 
> My main issue is that we won?t be using QString as a model for story text. String has many handy member functions that QLsit<uint> doesn?t. It?s also possible that QList<uint> is slower for many tasks.
> 
> But if everyone is fine with the change, we can make it work. Text search and nextCharacter() have to be rewritten anyway in order to handle non-latin text correctly.
> 
> The file format will probably use UTF8 anyway. Importing/exporting text will always handle character encoding as needed.

Jeans not convinced that we need to move to UTF32 at this stage in CTL, i.e. too early, not required. That we will see regressions in unknown places and find it hard to prove that the code is working properly for testing etc.

The assumption we?re only using the first half of UTF16 at this point would seem to cover our needs for now and doesn?t involve the changes describe, even if its a bad assumption to ignore the upper half. "UTF-32 is also supposed to use more than one unit for big code points, but in practical, it only requires one unit to store all code points of Unicode 6.0? http://unicodebook.readthedocs.io/unicode_encodings.html

I am a little concerned with the QString vs QList<uint> idea?. and speed. If its basic units and internal to StoryText are we better ending up with QByteArrays, basic arrays (or some other structure that doesn?t need manual memory management)?

"QString stores a string of 16-bit QChars, where each QChar corresponds one Unicode 4.0 character. (Unicode characters with code values above 65535 are stored using surrogate pairs, i.e., two consecutive Chars.)? - Does it really not deal with UTF32 properly or if its dealing with the surrogate pairs properly, do we really need to do more?.. 


Just some more 2c

Craig



From khaledhosny at eglug.org  Tue May 31 05:29:14 2016
From: khaledhosny at eglug.org (Khaled Hosny)
Date: Tue, 31 May 2016 07:29:14 +0200
Subject: [scribus-dev] Change to UTF-32
In-Reply-To: <2ADBFA59-CE7D-4EA0-BC60-C7E67CB3EAC3@scribus.info>
References: <016001d1ba87$b2ee0670$18ca1350$@googlemail.com>
 <3E21CCFD-A756-4D47-982D-4C691029AA21@scribus.info>
 <20160530191428.GA20467@macbook>
 <FC43BF8B-9CD6-438E-94B0-DDC2631D0941@scribus.info>
 <20160530193634.GB20467@macbook>
 <E176536F-ECA3-40B7-8CB0-03436B09F72E@scribus.info>
 <20160530200824.GD20467@macbook>
 <05A5F268-151B-40C0-8378-5997C3D5DA8A@scribus.info>
 <4C82221D-A19B-4DF9-9A2A-2D023E90A9DE@gmail.com>
 <2ADBFA59-CE7D-4EA0-BC60-C7E67CB3EAC3@scribus.info>
Message-ID: <20160531052914.GF20467@macbook>

On Tue, May 31, 2016 at 07:11:47AM +0200, Craig Bradney wrote:
> 
> > On 31 May 2016, at 00:41, Andreas Vox <andreas.vox at gmail.com> wrote:
> > 
> > 
> >> Am 30.05.2016 um 22:10 schrieb Craig Bradney <cbradney at scribus.info>:
> >> 
> >> 
> >>> On 30 May 2016, at 22:08, Khaled Hosny <khaledhosny at eglug.org> wrote:
> >>> 
> >>> On Mon, May 30, 2016 at 09:39:11PM +0200, Craig Bradney wrote:
> >>>> 
> >>>>> 
> >>>> 
> >>>> What about speed? We already need to speed up the text engine massively, including rendering (or especially rendering).
> >>> 
> >>> Converting a QString read from 25MB text file to UTF-32 takes <
> >>> 0.4 seconds, so that does not seem to be a concern. The current slowness
> >>> is mainly bad handling of text spanning multiple frames more than
> >>> anything else and Andreas is working on a fix for this.
> >>> 
> >>> 
> >> 
> >> 
> >> http://unicode.org/faq/utf_bom.html#UTF32 has some fairly strong wording about slow down from using UTF32.
> >> 
> > 
> > I wonder what they?d say if they saw our code in ScText_Shared?
> > 
> > As long as we pack every Char in an ScText struct, 8 vs. 16 vs. 32 bit doesn?t matter. A better structure would use a rope data structure or a list of strings for text and store style information as runs outside the text (coming soon).
> > 
> > My main issue is that we won?t be using QString as a model for story text. String has many handy member functions that QLsit<uint> doesn?t. It?s also possible that QList<uint> is slower for many tasks.
> > 
> > But if everyone is fine with the change, we can make it work. Text search and nextCharacter() have to be rewritten anyway in order to handle non-latin text correctly.
> > 
> > The file format will probably use UTF8 anyway. Importing/exporting text will always handle character encoding as needed.
> 
> Jeans not convinced that we need to move to UTF32 at this stage in
> CTL, i.e. too early, not required. That we will see regressions in
> unknown places and find it hard to prove that the code is working
> properly for testing etc.

The fact that we already have a few reports at GitHub about supporting
characters outside BMP says otherwise.

> The assumption we?re only using the first half of UTF16 at this point
> would seem to cover our needs for now and doesn?t involve the changes
> describe, even if its a bad assumption to ignore the upper half.
> "UTF-32 is also supposed to use more than one unit for big code
> points, but in practical, it only requires one unit to store all code
> points of Unicode 6.0?
> http://unicodebook.readthedocs.io/unicode_encodings.html

UTF-32 can have 2^32 characters, that is 4,294,967,296. There is no way
that Unicode will ever have that number of characters. By contrast
Unicode 9.0 (to be released yet) has only 128,172 characters.

> I am a little concerned with the QString vs QList<uint> idea?. and
> speed. If its basic units and internal to StoryText are we better
> ending up with QByteArrays, basic arrays (or some other structure that
> doesn?t need manual memory management)?

SoryText does not use QString even in trunk, it uses a QList of
something ScText which stores the characters as a QChar among other
things. What I did is replacing that QChar with uint and adapted the
code to work without.

Regards,
Khaled


From jghali at libertysurf.fr  Tue May 31 06:53:16 2016
From: jghali at libertysurf.fr (Jean Ghali)
Date: Tue, 31 May 2016 08:53:16 +0200
Subject: [scribus-dev] Change to UTF-32
In-Reply-To: <20160531052914.GF20467@macbook>
References: <016001d1ba87$b2ee0670$18ca1350$@googlemail.com>
 <3E21CCFD-A756-4D47-982D-4C691029AA21@scribus.info>
 <20160530191428.GA20467@macbook>
 <FC43BF8B-9CD6-438E-94B0-DDC2631D0941@scribus.info>
 <20160530193634.GB20467@macbook>
 <E176536F-ECA3-40B7-8CB0-03436B09F72E@scribus.info>
 <20160530200824.GD20467@macbook>
 <05A5F268-151B-40C0-8378-5997C3D5DA8A@scribus.info>
 <4C82221D-A19B-4DF9-9A2A-2D023E90A9DE@gmail.com>
 <2ADBFA59-CE7D-4EA0-BC60-C7E67CB3EAC3@scribus.info>
 <20160531052914.GF20467@macbook>
Message-ID: <d2e7a45d-cc32-5587-bb70-6c7ee74409c0@libertysurf.fr>

Le 31/05/2016 ? 07:29, Khaled Hosny a ?crit 
>> Jeans not convinced that we need to move to UTF32 at this stage in
>> CTL, i.e. too early, not required. That we will see regressions in
>> unknown places and find it hard to prove that the code is working
>> properly for testing etc.
> The fact that we already have a few reports at GitHub about supporting
> characters outside BMP says otherwise.

Supporting Emojis or historic scripts is certainly something nice, but supporting them is
not in any way required at this point. So I maintain my position. UTF-32 has advantages
but this is something which can wait after CTL code merge.



From williambader at hotmail.com  Tue May 31 09:36:59 2016
From: williambader at hotmail.com (William Bader)
Date: Tue, 31 May 2016 05:36:59 -0400
Subject: [scribus-dev] Change to UTF-32
In-Reply-To: <d2e7a45d-cc32-5587-bb70-6c7ee74409c0@libertysurf.fr>
References: <016001d1ba87$b2ee0670$18ca1350$@googlemail.com>,
 <3E21CCFD-A756-4D47-982D-4C691029AA21@scribus.info>,
 <20160530191428.GA20467@macbook>,
 <FC43BF8B-9CD6-438E-94B0-DDC2631D0941@scribus.info>,
 <20160530193634.GB20467@macbook>,
 <E176536F-ECA3-40B7-8CB0-03436B09F72E@scribus.info>,
 <20160530200824.GD20467@macbook>,
 <05A5F268-151B-40C0-8378-5997C3D5DA8A@scribus.info>,
 <4C82221D-A19B-4DF9-9A2A-2D023E90A9DE@gmail.com>,
 <2ADBFA59-CE7D-4EA0-BC60-C7E67CB3EAC3@scribus.info>,
 <20160531052914.GF20467@macbook>,
 <d2e7a45d-cc32-5587-bb70-6c7ee74409c0@libertysurf.fr>
Message-ID: <BLU179-W87243479985588A164899EC4460@phx.gbl>



> To: scribus-dev at lists.scribus.net
> From: jghali at libertysurf.fr
> Date: Tue, 31 May 2016 08:53:16 +0200
> Subject: Re: [scribus-dev] Change to UTF-32
> 
> Le 31/05/2016 ? 07:29, Khaled Hosny a ?crit 
> >> Jeans not convinced that we need to move to UTF32 at this stage in
> >> CTL, i.e. too early, not required. That we will see regressions in
> >> unknown places and find it hard to prove that the code is working
> >> properly for testing etc.
> > The fact that we already have a few reports at GitHub about supporting
> > characters outside BMP says otherwise.
> 
> Supporting Emojis or historic scripts is certainly something nice, but supporting them is
> not in any way required at this point. So I maintain my position. UTF-32 has advantages
> but this is something which can wait after CTL code merge.


Even with UTF-32, there are still combining characters https://en.wikipedia.org/wiki/Combining_character that require several code points per character. Doesn't that raise issues similar to characters outside BMP in UTF-16? Doing UTF-32 correctly might require touching all of the places that have BMP issues in UTF-16 because they will still need to deal with characters that require multiple integers, and Qt will be messier if you can't use QString.
William
 		 	   		  
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.scribus.net/pipermail/scribus-dev/attachments/20160531/2ce0d9ec/attachment.html>

From jghali at libertysurf.fr  Tue May 31 09:46:30 2016
From: jghali at libertysurf.fr (jghali at libertysurf.fr)
Date: Tue, 31 May 2016 05:46:30 -0400 (EDT)
Subject: [scribus-dev] Change to UTF-32
In-Reply-To: <BLU179-W87243479985588A164899EC4460@phx.gbl>
Message-ID: <763514557.298318156.1464687990095.JavaMail.root@zimbra48-e8.priv.proxad.net>



----- Mail original -----
> 
> > To: scribus-dev at lists.scribus.net
> > From: jghali at libertysurf.fr
> > Date: Tue, 31 May 2016 08:53:16 +0200
> > Subject: Re: [scribus-dev] Change to UTF-32
> > 
> > Le 31/05/2016 ? 07:29, Khaled Hosny a ?crit
> > >> Jeans not convinced that we need to move to UTF32 at this stage
> > >> in
> > >> CTL, i.e. too early, not required. That we will see regressions
> > >> in
> > >> unknown places and find it hard to prove that the code is
> > >> working
> > >> properly for testing etc.
> > > The fact that we already have a few reports at GitHub about
> > > supporting
> > > characters outside BMP says otherwise.
> > 
> > Supporting Emojis or historic scripts is certainly something nice,
> > but supporting them is
> > not in any way required at this point. So I maintain my position.
> > UTF-32 has advantages
> > but this is something which can wait after CTL code merge.
> 
> 
> 
> 
> Even with UTF-32, there are still combining characters
> https://en.wikipedia.org/wiki/Combining_character that require
> several code points per character. Doesn't that raise issues similar
> to characters outside BMP in UTF-16? Doing UTF-32 correctly might
> require touching all of the places that have BMP issues in UTF-16
> because they will still need to deal with characters that require
> multiple integers, and Qt will be messier if you can't use QString.
> 


Yep, I found a blog on this:
http://cpplover.blogspot.fr/2013/09/utf-32-is-not-fixed-legnth-character.html


From khaledhosny at eglug.org  Tue May 31 11:13:26 2016
From: khaledhosny at eglug.org (Khaled Hosny)
Date: Tue, 31 May 2016 13:13:26 +0200
Subject: [scribus-dev] Change to UTF-32
In-Reply-To: <BLU179-W87243479985588A164899EC4460@phx.gbl>
References: <FC43BF8B-9CD6-438E-94B0-DDC2631D0941@scribus.info>
 <20160530193634.GB20467@macbook>
 <E176536F-ECA3-40B7-8CB0-03436B09F72E@scribus.info>
 <20160530200824.GD20467@macbook>
 <05A5F268-151B-40C0-8378-5997C3D5DA8A@scribus.info>
 <4C82221D-A19B-4DF9-9A2A-2D023E90A9DE@gmail.com>
 <2ADBFA59-CE7D-4EA0-BC60-C7E67CB3EAC3@scribus.info>
 <20160531052914.GF20467@macbook>
 <d2e7a45d-cc32-5587-bb70-6c7ee74409c0@libertysurf.fr>
 <BLU179-W87243479985588A164899EC4460@phx.gbl>
Message-ID: <20160531111326.GG20467@macbook>

On Tue, May 31, 2016 at 05:36:59AM -0400, William Bader wrote:
> 
> 
> > To: scribus-dev at lists.scribus.net
> > From: jghali at libertysurf.fr
> > Date: Tue, 31 May 2016 08:53:16 +0200
> > Subject: Re: [scribus-dev] Change to UTF-32
> > 
> > Le 31/05/2016 ? 07:29, Khaled Hosny a ?crit 
> > >> Jeans not convinced that we need to move to UTF32 at this stage in
> > >> CTL, i.e. too early, not required. That we will see regressions in
> > >> unknown places and find it hard to prove that the code is working
> > >> properly for testing etc.
> > > The fact that we already have a few reports at GitHub about supporting
> > > characters outside BMP says otherwise.
> > 
> > Supporting Emojis or historic scripts is certainly something nice, but supporting them is
> > not in any way required at this point. So I maintain my position. UTF-32 has advantages
> > but this is something which can wait after CTL code merge.
> 
> 
> Even with UTF-32, there are still combining characters
> https://en.wikipedia.org/wiki/Combining_character that require several
> code points per character. Doesn't that raise issues similar to
> characters outside BMP in UTF-16? Doing UTF-32 correctly might require
> touching all of the places that have BMP issues in UTF-16 because they
> will still need to deal with characters that require multiple
> integers, and Qt will be messier if you can't use QString.

Combining characters is already handled such fine. We are not talking
here about user-perceived characters (grapheme clusters in Unicode
speak) but Unicode code points. Both UTF-8 and UTF-16 are variable size
encodings, so a code point can be made of one or more code units. The
implication of this is that if you process code units as if it were a
fixed length encoding you can end with illegal Unicode text, which is a
serious issue. With a fixed width encoding if you want to access the Nth
character you just access the Nth position in the array, with variable
length encodings you need to process the whole stream, and so on.

Any way, people are free to reject the parts of the patches they don?t
like. I don?t have this weird notion that some characters or scripts are
first class citizen and others we can think about them later. UTF-16 is
the worst of encodings (as I explained in the IRC discussion), it
encourages developers to write bad code, and the sorry state of non-BMP
in Scribus is a great testimony about this.

Regards,
Khaled


From scribus.user at gmail.com  Tue May 31 13:11:55 2016
From: scribus.user at gmail.com (Scribus User)
Date: Tue, 31 May 2016 08:11:55 -0500
Subject: [scribus-dev] Requesting testing of patch in #8753
Message-ID: <CAAt0FRgKT00ipX3ggvT6j=vTO=2q+5AvHweL8eKT==64k98j3A@mail.gmail.com>

Hey all,
William Bader was kind enough to submit a patch
https://bugs.scribus.net/view.php?id=8753 that allows triple edits of
images simultaneously in GIMP via scribus. Please test and report via to
let your results. Thanks!

Fyi, my laptop battery went kaput so I'm substantially off the radar these
days. Quite the limitation to test patches generously created by the
Community.

/Kunda
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.scribus.net/pipermail/scribus-dev/attachments/20160531/5bebcab1/attachment.html>

From gpittman at iglou.com  Tue May 31 13:30:13 2016
From: gpittman at iglou.com (Gregory Pittman)
Date: Tue, 31 May 2016 09:30:13 -0400
Subject: [scribus-dev] Change to UTF-32
In-Reply-To: <20160531111326.GG20467@macbook>
References: <FC43BF8B-9CD6-438E-94B0-DDC2631D0941@scribus.info>
 <20160530193634.GB20467@macbook>
 <E176536F-ECA3-40B7-8CB0-03436B09F72E@scribus.info>
 <20160530200824.GD20467@macbook>
 <05A5F268-151B-40C0-8378-5997C3D5DA8A@scribus.info>
 <4C82221D-A19B-4DF9-9A2A-2D023E90A9DE@gmail.com>
 <2ADBFA59-CE7D-4EA0-BC60-C7E67CB3EAC3@scribus.info>
 <20160531052914.GF20467@macbook>
 <d2e7a45d-cc32-5587-bb70-6c7ee74409c0@libertysurf.fr>
 <BLU179-W87243479985588A164899EC4460@phx.gbl>
 <20160531111326.GG20467@macbook>
Message-ID: <8d7164d4-210b-7da6-e1f1-33e8b0a3e55d@iglou.com>

On 05/31/2016 07:13 AM, Khaled Hosny wrote:

> Any way, people are free to reject the parts of the patches they don?t
> like. I don?t have this weird notion that some characters or scripts are
> first class citizen and others we can think about them later. UTF-16 is
> the worst of encodings (as I explained in the IRC discussion), it
> encourages developers to write bad code, and the sorry state of non-BMP
> in Scribus is a great testimony about this.
> 

I do think we should do our users the favor of making sure that the
included scripts are all still working before we release some change
like the character encoding. Everything should be fixable, but it makes
a bad impression to break things and then wait for people to complain.

Greg


From ale.comp_06 at xox.ch  Tue May 31 13:43:30 2016
From: ale.comp_06 at xox.ch (ale rimoldi)
Date: Tue, 31 May 2016 15:43:30 +0200
Subject: [scribus-dev] Change to UTF-32
In-Reply-To: <8d7164d4-210b-7da6-e1f1-33e8b0a3e55d@iglou.com>
References: <FC43BF8B-9CD6-438E-94B0-DDC2631D0941@scribus.info>
 <20160530193634.GB20467@macbook>
 <E176536F-ECA3-40B7-8CB0-03436B09F72E@scribus.info>
 <20160530200824.GD20467@macbook>
 <05A5F268-151B-40C0-8378-5997C3D5DA8A@scribus.info>
 <4C82221D-A19B-4DF9-9A2A-2D023E90A9DE@gmail.com>
 <2ADBFA59-CE7D-4EA0-BC60-C7E67CB3EAC3@scribus.info>
 <20160531052914.GF20467@macbook>
 <d2e7a45d-cc32-5587-bb70-6c7ee74409c0@libertysurf.fr>
 <BLU179-W87243479985588A164899EC4460@phx.gbl>
 <20160531111326.GG20467@macbook>
 <8d7164d4-210b-7da6-e1f1-33e8b0a3e55d@iglou.com>
Message-ID: <0e1e9c97-6d44-bad5-3758-39aa0a34c3bf@xox.ch>

hey

On 31/05/16 15:30, Gregory Pittman wrote:
> On 05/31/2016 07:13 AM, Khaled Hosny wrote:
>
>> Any way, people are free to reject the parts of the patches they don?t
>> like. I don?t have this weird notion that some characters or scripts are
>> first class citizen and others we can think about them later. UTF-16 is
>> the worst of encodings (as I explained in the IRC discussion), it
>> encourages developers to write bad code, and the sorry state of non-BMP
>> in Scribus is a great testimony about this.
>>
>
> I do think we should do our users the favor of making sure that the
> included scripts are all still working before we release some change
> like the character encoding. Everything should be fixable, but it makes
> a bad impression to break things and then wait for people to complain.

the problem are not the included scripts. we can fix them before 
including them.

the problem are all the other scripts that are in the wild.

but i'm not that worried. if we document how the included scripts had to 
be changed, at the time the new version will be stable, they have clear 
recipes for fixing their scripts.


ciao
a.l.e


From williambader at hotmail.com  Tue May 31 14:32:05 2016
From: williambader at hotmail.com (William Bader)
Date: Tue, 31 May 2016 10:32:05 -0400
Subject: [scribus-dev] Change to UTF-32
In-Reply-To: <0e1e9c97-6d44-bad5-3758-39aa0a34c3bf@xox.ch>
References: <FC43BF8B-9CD6-438E-94B0-DDC2631D0941@scribus.info>,
 <20160530193634.GB20467@macbook>,
 <E176536F-ECA3-40B7-8CB0-03436B09F72E@scribus.info>,
 <20160530200824.GD20467@macbook>,
 <05A5F268-151B-40C0-8378-5997C3D5DA8A@scribus.info>,
 <4C82221D-A19B-4DF9-9A2A-2D023E90A9DE@gmail.com>,
 <2ADBFA59-CE7D-4EA0-BC60-C7E67CB3EAC3@scribus.info>,
 <20160531052914.GF20467@macbook>,
 <d2e7a45d-cc32-5587-bb70-6c7ee74409c0@libertysurf.fr>,
 <BLU179-W87243479985588A164899EC4460@phx.gbl>,
 <20160531111326.GG20467@macbook>,
 <8d7164d4-210b-7da6-e1f1-33e8b0a3e55d@iglou.com>,
 <0e1e9c97-6d44-bad5-3758-39aa0a34c3bf@xox.ch>
Message-ID: <BLU179-W93C46B98EFF44C4FDF8EC0C4460@phx.gbl>



> To: scribus-dev at lists.scribus.net
> From: ale.comp_06 at xox.ch
> Date: Tue, 31 May 2016 15:43:30 +0200
> Subject: Re: [scribus-dev] Change to UTF-32
> 
> hey
> 
> On 31/05/16 15:30, Gregory Pittman wrote:
> > On 05/31/2016 07:13 AM, Khaled Hosny wrote:
> >
> >> Any way, people are free to reject the parts of the patches they don?t
> >> like. I don?t have this weird notion that some characters or scripts are
> >> first class citizen and others we can think about them later. UTF-16 is
> >> the worst of encodings (as I explained in the IRC discussion), it
> >> encourages developers to write bad code, and the sorry state of non-BMP
> >> in Scribus is a great testimony about this.
> >>
> >
> > I do think we should do our users the favor of making sure that the
> > included scripts are all still working before we release some change
> > like the character encoding. Everything should be fixable, but it makes
> > a bad impression to break things and then wait for people to complain.
> 
> the problem are not the included scripts. we can fix them before 
> including them.
> 
> the problem are all the other scripts that are in the wild.
> 
> but i'm not that worried. if we document how the included scripts had to 
> be changed, at the time the new version will be stable, they have clear 
> recipes for fixing their scripts.
> 
> 
> ciao
> a.l.e


Shouldn't the internal representation in Scribus be separate from the encoding used by the interface between python and Scribus? Python provides codecs for common encodings. https://docs.python.org/3/c-api/unicode.html  External formats like SLA files could remain UTF-8. What kinds of scripts would be broken?
William
 		 	   		  
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.scribus.net/pipermail/scribus-dev/attachments/20160531/9b1fc11b/attachment.html>

From ale.comp_06 at xox.ch  Tue May 31 14:33:35 2016
From: ale.comp_06 at xox.ch (ale rimoldi)
Date: Tue, 31 May 2016 16:33:35 +0200
Subject: [scribus-dev] Change to UTF-32
In-Reply-To: <BLU179-W93C46B98EFF44C4FDF8EC0C4460@phx.gbl>
References: <FC43BF8B-9CD6-438E-94B0-DDC2631D0941@scribus.info>
 <20160530193634.GB20467@macbook>
 <E176536F-ECA3-40B7-8CB0-03436B09F72E@scribus.info>
 <20160530200824.GD20467@macbook>
 <05A5F268-151B-40C0-8378-5997C3D5DA8A@scribus.info>
 <4C82221D-A19B-4DF9-9A2A-2D023E90A9DE@gmail.com>
 <2ADBFA59-CE7D-4EA0-BC60-C7E67CB3EAC3@scribus.info>
 <20160531052914.GF20467@macbook>
 <d2e7a45d-cc32-5587-bb70-6c7ee74409c0@libertysurf.fr>
 <BLU179-W87243479985588A164899EC4460@phx.gbl>
 <20160531111326.GG20467@macbook>
 <8d7164d4-210b-7da6-e1f1-33e8b0a3e55d@iglou.com>
 <0e1e9c97-6d44-bad5-3758-39aa0a34c3bf@xox.ch>
 <BLU179-W93C46B98EFF44C4FDF8EC0C4460@phx.gbl>
Message-ID: <75453edf-da11-9ffa-23f8-d33d472b2900@xox.ch>

hi

> Shouldn't the internal representation in Scribus be separate from the
> encoding used by the interface between python and Scribus? Python
> provides codecs for common
> encodings. https://docs.python.org/3/c-api/unicode.html  External
> formats like SLA files could remain UTF-8. What kinds of scripts would
> be broken?

no idea.

as said, i'm not worried...

ciao
a.l.e


From cbradney at scribus.info  Tue May 31 15:10:38 2016
From: cbradney at scribus.info (Craig Bradney)
Date: Tue, 31 May 2016 17:10:38 +0200
Subject: [scribus-dev] Change to UTF-32
In-Reply-To: <BLU179-W93C46B98EFF44C4FDF8EC0C4460@phx.gbl>
References: <FC43BF8B-9CD6-438E-94B0-DDC2631D0941@scribus.info>
 <20160530193634.GB20467@macbook>
 <E176536F-ECA3-40B7-8CB0-03436B09F72E@scribus.info>
 <20160530200824.GD20467@macbook>
 <05A5F268-151B-40C0-8378-5997C3D5DA8A@scribus.info>
 <4C82221D-A19B-4DF9-9A2A-2D023E90A9DE@gmail.com>
 <2ADBFA59-CE7D-4EA0-BC60-C7E67CB3EAC3@scribus.info>
 <20160531052914.GF20467@macbook>
 <d2e7a45d-cc32-5587-bb70-6c7ee74409c0@libertysurf.fr>
 <BLU179-W87243479985588A164899EC4460@phx.gbl>
 <20160531111326.GG20467@macbook>
 <8d7164d4-210b-7da6-e1f1-33e8b0a3e55d@iglou.com>
 <0e1e9c97-6d44-bad5-3758-39aa0a34c3bf@xox.ch>
 <BLU179-W93C46B98EFF44C4FDF8EC0C4460@phx.gbl>
Message-ID: <BD672A28-F068-46A8-8B3C-C6B1D852552C@scribus.info>


> On 31 May 2016, at 16:32, William Bader <williambader at hotmail.com> wrote:
> 
> 
> 
> > To: scribus-dev at lists.scribus.net <mailto:scribus-dev at lists.scribus.net>
> > From: ale.comp_06 at xox.ch <mailto:ale.comp_06 at xox.ch>
> > Date: Tue, 31 May 2016 15:43:30 +0200
> > Subject: Re: [scribus-dev] Change to UTF-32
> > 
> > hey
> > 
> > On 31/05/16 15:30, Gregory Pittman wrote:
> > > On 05/31/2016 07:13 AM, Khaled Hosny wrote:
> > >
> > >> Any way, people are free to reject the parts of the patches they don?t
> > >> like. I don?t have this weird notion that some characters or scripts are
> > >> first class citizen and others we can think about them later. UTF-16 is
> > >> the worst of encodings (as I explained in the IRC discussion), it
> > >> encourages developers to write bad code, and the sorry state of non-BMP
> > >> in Scribus is a great testimony about this.
> > >>
> > >
> > > I do think we should do our users the favor of making sure that the
> > > included scripts are all still working before we release some change
> > > like the character encoding. Everything should be fixable, but it makes
> > > a bad impression to break things and then wait for people to complain.
> > 
> > the problem are not the included scripts. we can fix them before 
> > including them.
> > 
> > the problem are all the other scripts that are in the wild.
> > 
> > but i'm not that worried. if we document how the included scripts had to 
> > be changed, at the time the new version will be stable, they have clear 
> > recipes for fixing their scripts.
> > 
> > 
> > ciao
> > a.l.e
> 
> 
> Shouldn't the internal representation in Scribus be separate from the encoding used by the interface between python and Scribus? Python provides codecs for common encodings. https://docs.python.org/3/c-api/unicode.html <https://docs.python.org/3/c-api/unicode.html>  External formats like SLA files could remain UTF-8. What kinds of scripts would be broken?
> 
> William
> 
> _


I agree, yes they should be.. 

Craig

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.scribus.net/pipermail/scribus-dev/attachments/20160531/e7b76111/attachment.html>

From cbradney at scribus.info  Tue May 31 17:20:14 2016
From: cbradney at scribus.info (Craig Bradney)
Date: Tue, 31 May 2016 19:20:14 +0200
Subject: [scribus-dev] Change to UTF-32
In-Reply-To: <20160531111326.GG20467@macbook>
References: <FC43BF8B-9CD6-438E-94B0-DDC2631D0941@scribus.info>
 <20160530193634.GB20467@macbook>
 <E176536F-ECA3-40B7-8CB0-03436B09F72E@scribus.info>
 <20160530200824.GD20467@macbook>
 <05A5F268-151B-40C0-8378-5997C3D5DA8A@scribus.info>
 <4C82221D-A19B-4DF9-9A2A-2D023E90A9DE@gmail.com>
 <2ADBFA59-CE7D-4EA0-BC60-C7E67CB3EAC3@scribus.info>
 <20160531052914.GF20467@macbook>
 <d2e7a45d-cc32-5587-bb70-6c7ee74409c0@libertysurf.fr>
 <BLU179-W87243479985588A164899EC4460@phx.gbl>
 <20160531111326.GG20467@macbook>
Message-ID: <02BB513F-77F6-41D0-8D71-7E517D1FE927@scribus.info>


> On 31 May 2016, at 13:13, Khaled Hosny <khaledhosny at eglug.org> wrote:
> 
> On Tue, May 31, 2016 at 05:36:59AM -0400, William Bader wrote:
>> 
>> 
>>> To: scribus-dev at lists.scribus.net
>>> From: jghali at libertysurf.fr
>>> Date: Tue, 31 May 2016 08:53:16 +0200
>>> Subject: Re: [scribus-dev] Change to UTF-32
>>> 
>>> Le 31/05/2016 ? 07:29, Khaled Hosny a ?crit 
>>>>> Jeans not convinced that we need to move to UTF32 at this stage in
>>>>> CTL, i.e. too early, not required. That we will see regressions in
>>>>> unknown places and find it hard to prove that the code is working
>>>>> properly for testing etc.
>>>> The fact that we already have a few reports at GitHub about supporting
>>>> characters outside BMP says otherwise.
>>> 
>>> Supporting Emojis or historic scripts is certainly something nice, but supporting them is
>>> not in any way required at this point. So I maintain my position. UTF-32 has advantages
>>> but this is something which can wait after CTL code merge.
>> 
>> 
>> Even with UTF-32, there are still combining characters
>> https://en.wikipedia.org/wiki/Combining_character that require several
>> code points per character. Doesn't that raise issues similar to
>> characters outside BMP in UTF-16? Doing UTF-32 correctly might require
>> touching all of the places that have BMP issues in UTF-16 because they
>> will still need to deal with characters that require multiple
>> integers, and Qt will be messier if you can't use QString.
> 
> Combining characters is already handled such fine. We are not talking
> here about user-perceived characters (grapheme clusters in Unicode
> speak) but Unicode code points. Both UTF-8 and UTF-16 are variable size
> encodings, so a code point can be made of one or more code units. The
> implication of this is that if you process code units as if it were a
> fixed length encoding you can end with illegal Unicode text, which is a
> serious issue. With a fixed width encoding if you want to access the Nth
> character you just access the Nth position in the array, with variable
> length encodings you need to process the whole stream, and so on.
> 
> Any way, people are free to reject the parts of the patches they don?t
> like. I don?t have this weird notion that some characters or scripts are
> first class citizen and others we can think about them later. UTF-16 is
> the worst of encodings (as I explained in the IRC discussion), it
> encourages developers to write bad code, and the sorry state of non-BMP
> in Scribus is a great testimony about this.
> 


Any ideas how we can break down the arguments for and against and come to a decision?

For me, the priority is to get CTL into trunk in a state that is as reliable as possible for existing functionality (i.e. Latin text) and extending to whatever scripts are already ok by CTL.

Maybe the UTF32 change is part of that, maybe not. If not, then the work should be either a) fix UTF16 handling or b) move to UTF32 as the step after merging.

Thoughts? Khaled? Avon? Jean?

Craig



From williambader at hotmail.com  Tue May 31 18:37:36 2016
From: williambader at hotmail.com (William Bader)
Date: Tue, 31 May 2016 14:37:36 -0400
Subject: [scribus-dev] Change to UTF-32
In-Reply-To: <02BB513F-77F6-41D0-8D71-7E517D1FE927@scribus.info>
References: <FC43BF8B-9CD6-438E-94B0-DDC2631D0941@scribus.info>,
 <20160530193634.GB20467@macbook>,
 <E176536F-ECA3-40B7-8CB0-03436B09F72E@scribus.info>,
 <20160530200824.GD20467@macbook>,
 <05A5F268-151B-40C0-8378-5997C3D5DA8A@scribus.info>,
 <4C82221D-A19B-4DF9-9A2A-2D023E90A9DE@gmail.com>,
 <2ADBFA59-CE7D-4EA0-BC60-C7E67CB3EAC3@scribus.info>,
 <20160531052914.GF20467@macbook>,
 <d2e7a45d-cc32-5587-bb70-6c7ee74409c0@libertysurf.fr>,
 <BLU179-W87243479985588A164899EC4460@phx.gbl>,
 <20160531111326.GG20467@macbook>,
 <02BB513F-77F6-41D0-8D71-7E517D1FE927@scribus.info>
Message-ID: <BLU179-W3461F06ACBA8CCCD677F66C4460@phx.gbl>



> From: cbradney at scribus.info
> Date: Tue, 31 May 2016 19:20:14 +0200
> To: scribus-dev at lists.scribus.net
> Subject: Re: [scribus-dev] Change to UTF-32
> 
> 
> > On 31 May 2016, at 13:13, Khaled Hosny <khaledhosny at eglug.org> wrote:
> > 
> > On Tue, May 31, 2016 at 05:36:59AM -0400, William Bader wrote:
> >> 
> >> 
> >>> To: scribus-dev at lists.scribus.net
> >>> From: jghali at libertysurf.fr
> >>> Date: Tue, 31 May 2016 08:53:16 +0200
> >>> Subject: Re: [scribus-dev] Change to UTF-32
> >>> 
> >>> Le 31/05/2016 ? 07:29, Khaled Hosny a ?crit 
> >>>>> Jeans not convinced that we need to move to UTF32 at this stage in
> >>>>> CTL, i.e. too early, not required. That we will see regressions in
> >>>>> unknown places and find it hard to prove that the code is working
> >>>>> properly for testing etc.
> >>>> The fact that we already have a few reports at GitHub about supporting
> >>>> characters outside BMP says otherwise.
> >>> 
> >>> Supporting Emojis or historic scripts is certainly something nice, but supporting them is
> >>> not in any way required at this point. So I maintain my position. UTF-32 has advantages
> >>> but this is something which can wait after CTL code merge.
> >> 
> >> 
> >> Even with UTF-32, there are still combining characters
> >> https://en.wikipedia.org/wiki/Combining_character that require several
> >> code points per character. Doesn't that raise issues similar to
> >> characters outside BMP in UTF-16? Doing UTF-32 correctly might require
> >> touching all of the places that have BMP issues in UTF-16 because they
> >> will still need to deal with characters that require multiple
> >> integers, and Qt will be messier if you can't use QString.
> > 
> > Combining characters is already handled such fine. We are not talking
> > here about user-perceived characters (grapheme clusters in Unicode
> > speak) but Unicode code points. Both UTF-8 and UTF-16 are variable size
> > encodings, so a code point can be made of one or more code units. The
> > implication of this is that if you process code units as if it were a
> > fixed length encoding you can end with illegal Unicode text, which is a
> > serious issue. With a fixed width encoding if you want to access the Nth
> > character you just access the Nth position in the array, with variable
> > length encodings you need to process the whole stream, and so on.
> > 
> > Any way, people are free to reject the parts of the patches they don?t
> > like. I don?t have this weird notion that some characters or scripts are
> > first class citizen and others we can think about them later. UTF-16 is
> > the worst of encodings (as I explained in the IRC discussion), it
> > encourages developers to write bad code, and the sorry state of non-BMP
> > in Scribus is a great testimony about this.
> > 
> 
> 
> Any ideas how we can break down the arguments for and against and come to a decision?
> 
> For me, the priority is to get CTL into trunk in a state that is as reliable as possible for existing functionality (i.e. Latin text) and extending to whatever scripts are already ok by CTL.
> 
> Maybe the UTF32 change is part of that, maybe not. If not, then the work should be either a) fix UTF16 handling or b) move to UTF32 as the step after merging.
> 
> Thoughts? Khaled? Avon? Jean?
> 
> Craig


I think that it is usually better to divide big changes into the smallest steps possible to make it easier to find bugs using bisection.
If Khaled's CTL code is mostly working and already depends on UTF32, though, it would probably be easier to take it at once instead of trying to split it into separate CTL and UTF32 patch sets.
If UTF32 is the eventual goal, I think that it isn't worth trying to fix UTF16 issues that would go away with the move to UTF32.
William


 		 	   		  
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.scribus.net/pipermail/scribus-dev/attachments/20160531/996c99b9/attachment.html>

From fahad.alsaidi at gmail.com  Tue May 31 18:52:38 2016
From: fahad.alsaidi at gmail.com (Fahad Al-Saidi)
Date: Tue, 31 May 2016 22:52:38 +0400
Subject: [scribus-dev] Change to UTF-32
In-Reply-To: <02BB513F-77F6-41D0-8D71-7E517D1FE927@scribus.info>
References: <FC43BF8B-9CD6-438E-94B0-DDC2631D0941@scribus.info>
 <20160530193634.GB20467@macbook>
 <E176536F-ECA3-40B7-8CB0-03436B09F72E@scribus.info>
 <20160530200824.GD20467@macbook>
 <05A5F268-151B-40C0-8378-5997C3D5DA8A@scribus.info>
 <4C82221D-A19B-4DF9-9A2A-2D023E90A9DE@gmail.com>
 <2ADBFA59-CE7D-4EA0-BC60-C7E67CB3EAC3@scribus.info>
 <20160531052914.GF20467@macbook>
 <d2e7a45d-cc32-5587-bb70-6c7ee74409c0@libertysurf.fr>
 <BLU179-W87243479985588A164899EC4460@phx.gbl>
 <20160531111326.GG20467@macbook>
 <02BB513F-77F6-41D0-8D71-7E517D1FE927@scribus.info>
Message-ID: <CAHJf0ggPZJq0so3fRmLa6pVeAUVXW9_riv-UMrmbaK+vfMMVgw@mail.gmail.com>

My suggestion is not merging the whole CTL branch at once. Then you can
revert any bad decision later on. The problem is CTL branch uses git while
main branch uses subsvn which makes it very tedious work to merge all
commits manually.
However, from my understanding, moving to UTF-32 is the easiest way to fix
current problems in CTL. Unless one have time and skills to fix the current
bugs and keep scribus using QChar, I tend to agree with Khaled in saving
time and effort by switching to UTF-32.

Now what we need is testing to see how it goes.

Cheers!
Fahad
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.scribus.net/pipermail/scribus-dev/attachments/20160531/cf6a5fec/attachment.html>

From cbradney at scribus.info  Tue May 31 20:12:28 2016
From: cbradney at scribus.info (Craig Bradney)
Date: Tue, 31 May 2016 22:12:28 +0200
Subject: [scribus-dev] Change to UTF-32
In-Reply-To: <CAHJf0ggPZJq0so3fRmLa6pVeAUVXW9_riv-UMrmbaK+vfMMVgw@mail.gmail.com>
References: <FC43BF8B-9CD6-438E-94B0-DDC2631D0941@scribus.info>
 <20160530193634.GB20467@macbook>
 <E176536F-ECA3-40B7-8CB0-03436B09F72E@scribus.info>
 <20160530200824.GD20467@macbook>
 <05A5F268-151B-40C0-8378-5997C3D5DA8A@scribus.info>
 <4C82221D-A19B-4DF9-9A2A-2D023E90A9DE@gmail.com>
 <2ADBFA59-CE7D-4EA0-BC60-C7E67CB3EAC3@scribus.info>
 <20160531052914.GF20467@macbook>
 <d2e7a45d-cc32-5587-bb70-6c7ee74409c0@libertysurf.fr>
 <BLU179-W87243479985588A164899EC4460@phx.gbl>
 <20160531111326.GG20467@macbook>
 <02BB513F-77F6-41D0-8D71-7E517D1FE927@scribus.info>
 <CAHJf0ggPZJq0so3fRmLa6pVeAUVXW9_riv-UMrmbaK+vfMMVgw@mail.gmail.com>
Message-ID: <F97274D6-EC42-4280-9AEC-5DB649EDA974@scribus.info>


> On 31 May 2016, at 20:52, Fahad Al-Saidi <fahad.alsaidi at gmail.com> wrote:
> 
> My suggestion is not merging the whole CTL branch at once. Then you can revert any bad decision later on. The problem is CTL branch uses git while main branch uses subsvn which makes it very tedious work to merge all commits manually.
> However, from my understanding, moving to UTF-32 is the easiest way to fix current problems in CTL. Unless one have time and skills to fix the current bugs and keep scribus using QChar, I tend to agree with Khaled in saving time and effort by switching to UTF-32.
> 
> Now what we need is testing to see how it goes.
> 


Anything I read comparing the various UTF encodings points either towards UTF8 and UTF32 being the right thing. UTF8 for reduced size and greater flexibility to expand as required, UTF32 for - in theory - fixed sizes. UTF16 just seems to be the evil opposite of the advantages of 8 and 32?

Craig

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.scribus.net/pipermail/scribus-dev/attachments/20160531/c33f60c9/attachment.html>

